2024-09-17T09:08:02.371 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T09:08:02.386 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T09:08:02.472 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908288 branch: reef description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7908288' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: reef ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - CEPHADM_AGENT_DOWN log-only-match: - CEPHADM_ sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 workunit: branch: reef sha1: 1cbdc9211df088cccbd94423c7cf3a2b85fe432b owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 5234 sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 sleep_before_teardown: 0 subset: 46/64 suite: orch suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_1cbdc9211df088cccbd94423c7cf3a2b85fe432b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1cbdc9211df088cccbd94423c7cf3a2b85fe432b targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGVFAiStxgVuklXdyb74rtJH/PmjPqpO9LeF6CpYqAP/D/es/rcLJxsyLWdRCFfDB75t2/qA0A59BRQlKJya14I= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNrUVa71OBHpyD0ZY0g+urNJZH9t6tePU8jVnCq458fbwTFRI9rsF2w8b2pUlI6Ti/kEeSYfI+widrbyuTcMj1g= tasks: - cephadm: allow_ptrace: false avoid_pacific_features: true cephadm_branch: v16.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v16.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_22:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T09:08:02.473 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1cbdc9211df088cccbd94423c7cf3a2b85fe432b/qa; will attempt to use it 2024-09-17T09:08:02.473 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1cbdc9211df088cccbd94423c7cf3a2b85fe432b/qa/tasks 2024-09-17T09:08:02.474 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T09:08:02.475 INFO:teuthology.task.internal:Checking packages... 2024-09-17T09:08:02.495 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9' 2024-09-17T09:08:02.495 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T09:08:02.495 INFO:teuthology.packaging:ref: None 2024-09-17T09:08:02.495 INFO:teuthology.packaging:tag: None 2024-09-17T09:08:02.496 INFO:teuthology.packaging:branch: reef 2024-09-17T09:08:02.496 INFO:teuthology.packaging:sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:08:02.496 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-17T09:08:02.745 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-855.g19e2a2be 2024-09-17T09:08:02.747 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T09:08:02.752 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T09:08:02.752 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T09:08:02.760 INFO:teuthology.task.internal:Saving configuration 2024-09-17T09:08:02.771 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T09:08:02.780 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T09:08:02.806 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908288', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 09:02:45.388036', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGVFAiStxgVuklXdyb74rtJH/PmjPqpO9LeF6CpYqAP/D/es/rcLJxsyLWdRCFfDB75t2/qA0A59BRQlKJya14I='} 2024-09-17T09:08:02.826 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908288', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 09:02:45.386635', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNrUVa71OBHpyD0ZY0g+urNJZH9t6tePU8jVnCq458fbwTFRI9rsF2w8b2pUlI6Ti/kEeSYfI+widrbyuTcMj1g='} 2024-09-17T09:08:02.827 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T09:08:02.834 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-17T09:08:02.834 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-17T09:08:02.834 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T09:08:02.895 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5012f03ac0>, signals=[15]) 2024-09-17T09:08:02.895 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T09:08:02.902 INFO:teuthology.task.internal:Opening connections... 2024-09-17T09:08:02.902 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-09-17T09:08:02.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T09:08:02.977 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-09-17T09:08:02.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T09:08:03.053 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T09:08:03.060 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-09-17T09:08:03.079 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-09-17T09:08:03.079 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-09-17T09:08:03.137 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-09-17T09:08:03.138 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-09-17T09:08:03.138 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-09-17T09:08:03.138 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-09-17T09:08:03.138 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-09-17T09:08:03.138 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-09-17T09:08:03.138 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T09:08:03.138 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-09-17T09:08:03.138 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-09-17T09:08:03.139 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T09:08:03.139 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-09-17T09:08:03.139 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T09:08:03.139 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T09:08:03.139 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T09:08:03.139 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-09-17T09:08:03.159 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-09-17T09:08:03.177 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-09-17T09:08:03.178 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-09-17T09:08:03.236 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-09-17T09:08:03.236 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-09-17T09:08:03.236 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-09-17T09:08:03.236 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-09-17T09:08:03.236 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-09-17T09:08:03.236 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-09-17T09:08:03.236 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T09:08:03.236 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-09-17T09:08:03.236 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-09-17T09:08:03.236 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T09:08:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-09-17T09:08:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T09:08:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T09:08:03.237 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T09:08:03.237 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-09-17T09:08:03.258 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T09:08:03.269 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T09:08:03.275 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T09:08:03.276 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-09-17T09:08:03.278 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-09-17T09:08:03.295 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T09:08:03.301 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T09:08:03.302 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-09-17T09:08:03.337 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-09-17T09:08:03.487 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T09:08:03.539 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T09:08:03.553 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T09:08:03.553 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T09:08:03.553 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T09:08:03.554 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T09:08:03.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T09:08:03.570 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-09-17T09:08:03.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T09:08:03.571 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-09-17T09:08:03.628 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-503.el9.x86_64 2024-09-17T09:08:03.628 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-503.el9.x86_64 2024-09-17T09:08:03.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-09-17T09:08:03.630 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-503.el9.x86_64 2024-09-17T09:08:03.630 INFO:teuthology.task.kernel:Running kernel on smithi103: 5.14.0-503.el9.x86_64 2024-09-17T09:08:03.630 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum install -y kernel 2024-09-17T09:08:07.388 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.3 MB 00:01 2024-09-17T09:08:09.369 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.3 MB 00:02 2024-09-17T09:08:11.379 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-17T09:08:12.922 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-17T09:08:16.641 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.5 MB 00:01 2024-09-17T09:08:18.093 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 101 kB/s | 19 kB 00:00 2024-09-17T09:08:18.548 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.5 MB 00:01 2024-09-17T09:08:19.431 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-17T09:08:19.974 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 93 kB/s | 19 kB 00:00 2024-09-17T09:08:20.728 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-17T09:08:25.080 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-17T09:08:26.167 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-17T09:08:27.089 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T09:08:27.090 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T09:08:27.196 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-09-17T09:08:27.212 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-09-17T09:08:27.212 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-09-17T09:08:27.212 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-09-17T09:08:27.212 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-09-17T09:08:27.212 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T09:08:27.212 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-09-17T09:08:27.212 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T09:08:27.212 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T09:08:27.212 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T09:08:27.213 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:08:27.213 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-09-17T09:08:27.213 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-09-17T09:08:27.213 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-09-17T09:08:27.213 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:08:27.213 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 79 M 2024-09-17T09:08:27.214 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 126 M 2024-09-17T09:08:27.214 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-09-17T09:08:27.335 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 478 kB/s | 36 kB 00:00 2024-09-17T09:08:28.073 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T09:08:28.074 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T09:08:28.155 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-09-17T09:08:28.167 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-17T09:08:28.168 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-09-17T09:08:28.168 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-17T09:08:28.168 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-09-17T09:08:28.168 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T09:08:28.168 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-09-17T09:08:28.168 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T09:08:28.168 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T09:08:28.168 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T09:08:28.168 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-17T09:08:28.168 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-09-17T09:08:28.168 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-09-17T09:08:28.168 INFO:teuthology.orchestra.run.smithi103.stdout:Install 4 Packages 2024-09-17T09:08:28.169 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-17T09:08:28.169 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 79 M 2024-09-17T09:08:28.169 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 126 M 2024-09-17T09:08:28.169 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-09-17T09:08:28.419 INFO:teuthology.orchestra.run.smithi103.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 296 kB/s | 36 kB 00:00 2024-09-17T09:08:28.610 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 26 MB/s | 35 MB 00:01 2024-09-17T09:08:28.869 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 9.7 MB/s | 16 MB 00:01 2024-09-17T09:08:29.394 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 14 MB/s | 29 MB 00:02 2024-09-17T09:08:29.395 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:08:29.395 INFO:teuthology.orchestra.run.smithi029.stdout:Total 36 MB/s | 79 MB 00:02 2024-09-17T09:08:30.109 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-09-17T09:08:30.526 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-09-17T09:08:30.527 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-09-17T09:08:30.927 INFO:teuthology.orchestra.run.smithi103.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 6.0 MB/s | 16 MB 00:02 2024-09-17T09:08:31.500 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-09-17T09:08:31.501 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-09-17T09:08:32.911 INFO:teuthology.orchestra.run.smithi103.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 6.4 MB/s | 29 MB 00:04 2024-09-17T09:08:34.062 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-09-17T09:08:34.661 INFO:teuthology.orchestra.run.smithi103.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 5.4 MB/s | 35 MB 00:06 2024-09-17T09:08:34.661 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-09-17T09:08:34.661 INFO:teuthology.orchestra.run.smithi103.stdout:Total 12 MB/s | 79 MB 00:06 2024-09-17T09:08:34.803 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T09:08:34.908 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T09:08:35.345 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-09-17T09:08:35.694 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-09-17T09:08:35.694 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-09-17T09:08:36.630 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-09-17T09:08:36.631 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-09-17T09:08:37.661 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T09:08:37.869 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T09:08:39.077 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-09-17T09:08:39.776 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T09:08:39.872 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T09:08:42.643 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T09:08:42.782 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T09:08:42.995 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T09:08:43.059 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T09:08:47.630 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T09:08:47.833 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T09:08:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T09:08:52.696 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T09:09:29.709 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T09:09:29.749 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T09:09:30.215 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T09:09:30.215 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T09:09:30.215 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T09:09:30.215 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T09:09:31.267 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T09:09:31.267 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:09:31.267 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-09-17T09:09:31.267 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T09:09:31.268 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T09:09:31.268 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T09:09:31.268 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T09:09:31.268 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:09:31.268 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-09-17T09:09:31.383 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T09:09:31.427 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T09:09:31.718 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T09:09:31.791 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T09:09:31.791 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T09:09:31.791 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T09:09:31.791 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T09:09:31.826 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T09:09:31.826 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T09:09:31.826 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T09:09:31.826 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T09:09:31.826 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T09:09:31.826 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T09:09:31.827 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T09:09:31.827 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T09:09:31.886 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T09:09:31.887 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T09:09:31.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2024-09-17T09:09:31.918 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2024-09-17T09:09:31.918 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2024-09-17T09:09:31.918 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2024-09-17T09:09:31.918 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 82.el9 2024-09-17T09:09:31.918 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2024-09-17T09:09:31.918 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T09:09:31.918 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2024-09-17T09:09:31.919 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8274670 2024-09-17T09:09:31.919 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2024-09-17T09:09:31.919 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T09:09:31.919 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T09:09:31.919 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T09:09:31.919 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T09:09:31.919 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2024-09-17T09:09:31.919 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2024-09-17T09:09:31.919 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T09:09:31.919 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2024-09-17T09:09:31.920 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2024-09-17T09:09:31.920 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:09:31.920 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T09:09:31.920 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T09:09:31.920 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T09:09:31.920 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2024-09-17T09:09:31.921 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:09:31.921 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T09:09:31.922 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T09:09:31.922 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-508.el9.x86_64 2024-09-17T09:09:31.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T09:09:32.623 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T09:09:32.624 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-17T09:09:32.624 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-09-17T09:09:32.624 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T09:09:32.624 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T09:09:32.624 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T09:09:32.625 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T09:09:32.625 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-17T09:09:32.625 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-09-17T09:09:32.671 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-09-17T09:09:32.973 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T09:09:33.056 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T09:09:33.056 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T09:09:33.056 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T09:09:33.056 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T09:09:33.056 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T09:09:33.056 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-17T09:09:33.056 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T09:09:33.057 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T09:09:33.115 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T09:09:33.115 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T09:09:33.115 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-09-17T09:09:33.148 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-09-17T09:09:33.148 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-09-17T09:09:33.148 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.06 2024-09-17T09:09:33.148 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 82.el9 2024-09-17T09:09:33.148 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-09-17T09:09:33.148 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T09:09:33.149 INFO:teuthology.orchestra.run.smithi103.stdout:Group : Unspecified 2024-09-17T09:09:33.149 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 8274670 2024-09-17T09:09:33.149 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-09-17T09:09:33.149 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T09:09:33.149 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T09:09:33.149 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T09:09:33.149 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T09:09:33.150 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : builder@centos.org 2024-09-17T09:09:33.150 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-09-17T09:09:33.150 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T09:09:33.150 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-09-17T09:09:33.150 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-09-17T09:09:33.150 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-17T09:09:33.150 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T09:09:33.151 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T09:09:33.151 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T09:09:33.151 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-09-17T09:09:33.151 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-17T09:09:33.151 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T09:09:33.153 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T09:09:33.153 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 5.14.0-508.el9.x86_64 2024-09-17T09:09:33.153 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T09:09:33.766 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-09-17T09:09:34.189 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T09:09:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-09-17T09:09:34.237 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-09-17T09:09:34.252 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.KME4AIUQVX 2024-09-17T09:09:34.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KME4AIUQVX 2024-09-17T09:09:34.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.KME4AIUQVX 2024-09-17T09:09:34.487 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.KME4AIUQVX is 6KB 2024-09-17T09:09:34.537 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.KME4AIUQVX 2024-09-17T09:09:34.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T09:09:34.619 INFO:teuthology.orchestra.run.smithi029.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T09:09:34.619 INFO:teuthology.orchestra.run.smithi029.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T09:09:34.620 INFO:teuthology.orchestra.run.smithi029.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T09:09:34.620 INFO:teuthology.orchestra.run.smithi029.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T09:09:34.621 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T09:09:34.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-09-17T09:09:35.147 INFO:teuthology.orchestra.run.smithi103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T09:09:35.183 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-09-17T09:09:35.185 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-09-17T09:09:35.201 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.SUv3lHYvPl 2024-09-17T09:09:35.202 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SUv3lHYvPl 2024-09-17T09:09:35.269 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.SUv3lHYvPl 2024-09-17T09:09:35.404 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.SUv3lHYvPl is 6KB 2024-09-17T09:09:35.452 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.SUv3lHYvPl 2024-09-17T09:09:35.467 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T09:09:35.530 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T09:09:35.530 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T09:09:35.531 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T09:09:35.531 INFO:teuthology.orchestra.run.smithi103.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T09:09:35.532 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T09:09:35.675 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-09-17T09:10:04.796 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T09:10:04.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-17T09:10:04.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T09:10:05.679 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-17T09:10:05.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-17T09:10:05.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T09:10:23.293 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-09-17T09:10:24.065 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-09-17T09:10:32.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-17T09:10:32.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T09:10:33.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-17T09:10:33.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T09:10:36.125 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-09-17T09:10:48.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-17T09:10:48.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T09:10:51.197 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-09-17T09:11:06.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-17T09:11:06.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T09:11:32.330 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T09:11:38.252 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-09-17T09:11:38.600 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-17T09:11:38.601 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-17T09:11:38.601 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-09-17T09:11:38.617 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-508.el9.x86_64 2024-09-17T09:11:38.617 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T09:11:38.617 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T09:11:38.617 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T09:11:39.618 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-17T09:11:39.618 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T09:11:39.686 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2024-09-17T09:11:39.710 DEBUG:teuthology.parallel:result is None 2024-09-17T09:11:44.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-17T09:11:44.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T09:11:44.687 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-09-17T09:11:45.086 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-17T09:11:45.087 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T09:11:45.087 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-09-17T09:11:45.104 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-508.el9.x86_64 2024-09-17T09:11:45.104 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T09:11:45.104 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T09:11:45.104 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T09:11:46.105 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T09:11:46.105 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T09:11:46.180 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-09-17T09:11:46.209 DEBUG:teuthology.parallel:result is None 2024-09-17T09:11:46.209 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T09:11:46.218 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T09:11:46.219 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T09:11:46.221 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T09:11:46.240 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T09:11:46.246 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T09:11:46.253 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T09:11:46.253 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T09:11:46.280 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T09:11:46.340 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T09:11:46.348 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T09:11:46.348 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T09:11:46.350 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T09:11:46.381 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T09:11:46.387 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T09:11:46.393 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T09:11:46.401 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T09:11:46.401 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T09:11:46.408 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T09:11:46.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T09:11:46.438 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T09:11:46.469 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T09:11:46.478 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T09:11:46.479 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T09:11:46.508 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T09:11:46.525 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T09:11:46.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T09:11:46.660 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T09:11:46.741 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T09:11:46.788 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-17T09:11:46.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T09:11:46.856 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T09:11:46.899 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T09:11:46.956 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T09:11:47.040 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T09:11:47.085 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-17T09:11:47.085 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T09:11:47.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-09-17T09:11:47.156 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-09-17T09:11:47.202 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T09:11:47.262 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T09:11:47.585 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T09:11:47.592 INFO:teuthology.task.internal:Starting timer... 2024-09-17T09:11:47.593 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T09:11:47.602 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T09:11:47.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-09-17T09:11:47.701 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-09-17T09:11:47.703 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-09-17T09:11:47.770 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-09-17T09:11:47.772 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T09:11:47.772 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-09-17T09:11:47.801 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-09-17T09:11:47.801 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-09-17T09:11:47.829 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-09-17T09:11:47.829 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive'} 2024-09-17T09:11:47.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T09:11:47.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T09:11:47.873 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T09:11:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T09:11:47.901 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T09:11:47.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-09-17T09:11:47.948 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-09-17T09:11:47.979 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T09:11:47.989 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T09:11:48.002 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T09:11:48.003 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi103.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T09:15:40.526 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com')] 2024-09-17T09:15:40.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-17T09:15:40.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T09:15:40.609 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-09-17T09:15:40.686 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-17T09:15:40.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-17T09:15:40.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T09:15:40.764 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-09-17T09:15:40.849 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-09-17T09:15:40.849 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T09:15:40.859 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T09:15:40.860 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T09:15:40.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T09:15:40.863 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T09:15:40.863 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T09:15:40.892 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T09:15:40.908 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T09:15:40.920 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T09:15:40.935 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-09-17T09:15:40.935 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T09:15:40.947 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-09-17T09:15:40.960 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-09-17T09:15:40.962 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T09:15:40.972 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-09-17T09:15:40.977 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T09:15:40.986 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T09:15:41.000 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T09:15:41.034 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-09-17T09:15:41.037 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T09:15:41.037 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-09-17T09:15:41.037 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T09:15:41.037 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T09:15:41.037 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T09:15:41.037 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T09:15:41.057 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-09-17T09:15:41.061 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T09:15:41.061 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-09-17T09:15:41.061 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T09:15:41.061 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T09:15:41.061 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T09:15:41.061 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T09:15:41.061 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-17T09:15:41.150 INFO:tasks.cephadm:Config: {'allow_ptrace': False, 'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v16.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'CEPHADM_AGENT_DOWN'], 'log-only-match': ['CEPHADM_'], 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'} 2024-09-17T09:15:41.150 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v16.2.0 2024-09-17T09:15:41.150 INFO:tasks.cephadm:Cluster fsid is 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:15:41.151 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-17T09:15:41.151 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.29', 'mon.c': '[v2:172.21.15.29:3301,v1:172.21.15.29:6790]', 'mon.b': '172.21.15.103'} 2024-09-17T09:15:41.151 INFO:tasks.cephadm:First mon is mon.a on smithi029 2024-09-17T09:15:41.151 INFO:tasks.cephadm:First mgr is y 2024-09-17T09:15:41.151 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-17T09:15:41.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2024-09-17T09:15:41.179 DEBUG:teuthology.orchestra.run.smithi103:> sudo hostname $(hostname -s) 2024-09-17T09:15:41.204 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.0)... 2024-09-17T09:15:41.204 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T09:15:41.423 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r--. 1 ubuntu ubuntu 288386 Sep 17 09:15 /home/ubuntu/cephtest/cephadm 2024-09-17T09:15:41.424 DEBUG:teuthology.orchestra.run.smithi103:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T09:15:41.625 INFO:teuthology.orchestra.run.smithi103.stdout:-rw-r--r--. 1 ubuntu ubuntu 288386 Sep 17 09:15 /home/ubuntu/cephtest/cephadm 2024-09-17T09:15:41.625 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T09:15:41.643 DEBUG:teuthology.orchestra.run.smithi103:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T09:15:41.669 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v16.2.0 on all hosts... 2024-09-17T09:15:41.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 pull 2024-09-17T09:15:41.685 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 pull 2024-09-17T09:15:42.563 INFO:teuthology.orchestra.run.smithi103.stderr:Pulling container image quay.io/ceph/ceph:v16.2.0... 2024-09-17T09:15:43.509 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image quay.io/ceph/ceph:v16.2.0... 2024-09-17T09:15:54.370 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-09-17T09:15:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_version": "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)", 2024-09-17T09:15:54.370 INFO:teuthology.orchestra.run.smithi103.stdout: "image_id": "24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15", 2024-09-17T09:15:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: "repo_digests": [ 2024-09-17T09:15:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: "quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000", 2024-09-17T09:15:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: "quay.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a" 2024-09-17T09:15:54.371 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-09-17T09:15:54.371 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-09-17T09:15:56.048 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:15:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)", 2024-09-17T09:15:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "image_id": "24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15", 2024-09-17T09:15:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "repo_digests": [ 2024-09-17T09:15:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000", 2024-09-17T09:15:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "quay.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a" 2024-09-17T09:15:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-17T09:15:56.048 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:15:56.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2024-09-17T09:15:56.119 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph 2024-09-17T09:15:56.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2024-09-17T09:15:56.177 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /etc/ceph 2024-09-17T09:15:56.221 INFO:tasks.cephadm:Writing seed config... 2024-09-17T09:15:56.222 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-17T09:15:56.222 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-17T09:15:56.223 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-17T09:15:56.223 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-09-17T09:15:56.223 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-17T09:15:56.223 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-17T09:15:56.223 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-17T09:15:56.223 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-17T09:15:56.223 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-17T09:15:56.224 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-17T09:15:56.224 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-17T09:15:56.239 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 69135c5a-74d5-11ef-bceb-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-17T09:15:56.239 DEBUG:teuthology.orchestra.run.smithi029:mon.a> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.a.service 2024-09-17T09:15:56.281 DEBUG:teuthology.orchestra.run.smithi029:mgr.y> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.y.service 2024-09-17T09:15:56.324 INFO:tasks.cephadm:Bootstrapping... 2024-09-17T09:15:56.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 -v bootstrap --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.29 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:15:56.516 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman --version 2024-09-17T09:15:56.536 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: podman version 5.2.2 2024-09-17T09:15:56.538 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying podman|docker is present... 2024-09-17T09:15:56.538 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman --version 2024-09-17T09:15:56.556 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: podman version 5.2.2 2024-09-17T09:15:56.558 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying lvm2 is present... 2024-09-17T09:15:56.558 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying time synchronization is in place... 2024-09-17T09:15:56.558 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled chrony.service 2024-09-17T09:15:56.565 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T09:15:56.566 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active chrony.service 2024-09-17T09:15:56.572 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2024-09-17T09:15:56.572 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled chronyd.service 2024-09-17T09:15:56.579 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: enabled 2024-09-17T09:15:56.580 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active chronyd.service 2024-09-17T09:15:56.586 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: active 2024-09-17T09:15:56.586 INFO:teuthology.orchestra.run.smithi029.stderr:Unit chronyd.service is enabled and running 2024-09-17T09:15:56.586 INFO:teuthology.orchestra.run.smithi029.stderr:Repeating the final host check... 2024-09-17T09:15:56.587 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman --version 2024-09-17T09:15:56.606 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: podman version 5.2.2 2024-09-17T09:15:56.607 INFO:teuthology.orchestra.run.smithi029.stderr:podman|docker (/bin/podman) is present 2024-09-17T09:15:56.607 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl is present 2024-09-17T09:15:56.607 INFO:teuthology.orchestra.run.smithi029.stderr:lvcreate is present 2024-09-17T09:15:56.608 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled chrony.service 2024-09-17T09:15:56.614 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T09:15:56.614 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active chrony.service 2024-09-17T09:15:56.620 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2024-09-17T09:15:56.621 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled chronyd.service 2024-09-17T09:15:56.628 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: enabled 2024-09-17T09:15:56.628 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active chronyd.service 2024-09-17T09:15:56.634 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: active 2024-09-17T09:15:56.635 INFO:teuthology.orchestra.run.smithi029.stderr:Unit chronyd.service is enabled and running 2024-09-17T09:15:56.635 INFO:teuthology.orchestra.run.smithi029.stderr:Host looks OK 2024-09-17T09:15:56.635 INFO:teuthology.orchestra.run.smithi029.stderr:Cluster fsid: 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:15:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:Acquiring lock 139727085512736 on /run/cephadm/69135c5a-74d5-11ef-bceb-c7b262605968.lock 2024-09-17T09:15:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:Lock 139727085512736 acquired on /run/cephadm/69135c5a-74d5-11ef-bceb-c7b262605968.lock 2024-09-17T09:15:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying IP 172.21.15.29 port 3300 ... 2024-09-17T09:15:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying IP 172.21.15.29 port 6789 ... 2024-09-17T09:15:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:Base mon IP is 172.21.15.29, final addrv is [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-09-17T09:15:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /sbin/ip route ls 2024-09-17T09:15:56.639 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.29 metric 100 2024-09-17T09:15:56.639 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.29 metric 100 2024-09-17T09:15:56.639 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /sbin/ip -6 route ls 2024-09-17T09:15:56.642 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-09-17T09:15:56.642 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-17T09:15:56.642 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1790sec hoplimit 64 pref medium 2024-09-17T09:15:56.642 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /sbin/ip -6 addr ls 2024-09-17T09:15:56.644 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-17T09:15:56.645 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-09-17T09:15:56.645 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-09-17T09:15:56.645 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-17T09:15:56.645 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2024-09-17T09:15:56.645 INFO:teuthology.orchestra.run.smithi029.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-09-17T09:15:56.645 INFO:teuthology.orchestra.run.smithi029.stderr:Mon IP 172.21.15.29 is in CIDR network 172.21.0.0/20 2024-09-17T09:15:56.646 INFO:teuthology.orchestra.run.smithi029.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-17T09:15:56.646 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image quay.io/ceph/ceph:v16.2.0... 2024-09-17T09:15:56.647 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman pull quay.io/ceph/ceph:v16.2.0 2024-09-17T09:15:56.751 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v16.2.0... 2024-09-17T09:15:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Getting image source signatures 2024-09-17T09:15:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Copying blob sha256:958e3c20ad47ec7f3ec0446a4a630f2285dd6b0a4c969f4f702f7c2e78f3d68b 2024-09-17T09:15:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-09-17T09:15:57.149 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Copying config sha256:24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15 2024-09-17T09:15:57.361 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: Writing manifest to image destination 2024-09-17T09:15:57.515 INFO:teuthology.orchestra.run.smithi029.stderr:/bin/podman: 24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15 2024-09-17T09:15:57.518 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --version 2024-09-17T09:15:57.932 INFO:teuthology.orchestra.run.smithi029.stderr:ceph: ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable) 2024-09-17T09:15:58.472 INFO:teuthology.orchestra.run.smithi029.stderr:Ceph version: ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable) 2024-09-17T09:15:58.473 INFO:teuthology.orchestra.run.smithi029.stderr:Extracting ceph user uid/gid from container image... 2024-09-17T09:15:58.473 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 -c %u %g /var/lib/ceph 2024-09-17T09:15:58.866 INFO:teuthology.orchestra.run.smithi029.stderr:stat: 167 167 2024-09-17T09:15:59.452 INFO:teuthology.orchestra.run.smithi029.stderr:Creating initial keys... 2024-09-17T09:15:59.452 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --gen-print-key 2024-09-17T09:15:59.880 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-authtool: AQDPSOlmp//SLhAAjXPPG1qewQYCtA3RKVsqUQ== 2024-09-17T09:16:02.641 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --gen-print-key 2024-09-17T09:16:08.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-authtool: AQDXSOlmp3avHRAAbe2Qc46nkuXfFF4/QxeQhg== 2024-09-17T09:16:17.507 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16.2.0 --gen-print-key 2024-09-17T09:16:17.893 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-authtool: AQDhSOlmqJesMBAAI8oDJoDPLxI2AINSuW0S5Q== 2024-09-17T09:16:18.394 INFO:teuthology.orchestra.run.smithi029.stderr:Creating initial monmap... 2024-09-17T09:16:18.395 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmp2_p3ukqv:/tmp/monmap:z quay.io/ceph/ceph:v16.2.0 --create --clobber --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 --addv a [v2:172.21.15.29:3300,v1:172.21.15.29:6789] /tmp/monmap 2024-09-17T09:16:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T09:16:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:16:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T09:16:19.291 INFO:teuthology.orchestra.run.smithi029.stderr:monmaptool for a [v2:172.21.15.29:3300,v1:172.21.15.29:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T09:16:19.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: set fsid to 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:16:19.291 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T09:16:19.291 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-09-17T09:16:19.291 INFO:teuthology.orchestra.run.smithi029.stderr:Creating mon... 2024-09-17T09:16:19.292 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmp4d9z6r6r:/tmp/keyring:z -v /tmp/ceph-tmp2_p3ukqv:/tmp/monmap:z quay.io/ceph/ceph:v16.2.0 --mkfs -i a --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-09-17T09:16:19.727 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.651+0000 7f5a96214700 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-17T09:16:19.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.652+0000 7f5a96214700 1 imported monmap: 2024-09-17T09:16:19.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: epoch 0 2024-09-17T09:16:19.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: fsid 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:16:19.728 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: last_changed 2024-09-17T09:16:18.701305+0000 2024-09-17T09:16:19.753 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: created 2024-09-17T09:16:18.701305+0000 2024-09-17T09:16:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-09-17T09:16:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-09-17T09:16:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-09-17T09:16:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.652+0000 7f5a96214700 0 /usr/bin/ceph-mon: set fsid to 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:16:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: RocksDB version: 6.8.1 2024-09-17T09:16:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-17T09:16:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Compile date Mar 30 2021 2024-09-17T09:16:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: DB SUMMARY 2024-09-17T09:16:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-09-17T09:16:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-09-17T09:16:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.error_if_exists: 0 2024-09-17T09:16:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.create_if_missing: 1 2024-09-17T09:16:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.paranoid_checks: 1 2024-09-17T09:16:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.env: 0x5624dfb9d1c0 2024-09-17T09:16:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.fs: Posix File System 2024-09-17T09:16:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.info_log: 0x5624e23eeb80 2024-09-17T09:16:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-17T09:16:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.statistics: (nil) 2024-09-17T09:16:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.use_fsync: 0 2024-09-17T09:16:19.756 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.max_log_file_size: 0 2024-09-17T09:16:19.757 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T09:16:19.757 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T09:16:19.757 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-17T09:16:19.757 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-17T09:16:19.757 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.allow_fallocate: 1 2024-09-17T09:16:19.757 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-17T09:16:19.758 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-17T09:16:19.758 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.use_direct_reads: 0 2024-09-17T09:16:19.758 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T09:16:19.758 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.create_missing_column_families: 0 2024-09-17T09:16:19.758 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.db_log_dir: 2024-09-17T09:16:19.758 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-17T09:16:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T09:16:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.max_subcompactions: 1 2024-09-17T09:16:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.max_background_flushes: -1 2024-09-17T09:16:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T09:16:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T09:16:19.759 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T09:16:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T09:16:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T09:16:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.advise_random_on_open: 1 2024-09-17T09:16:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-17T09:16:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.write_buffer_manager: 0x5624e23f9410 2024-09-17T09:16:19.760 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T09:16:19.761 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-17T09:16:19.761 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T09:16:19.761 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T09:16:19.761 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.rate_limiter: (nil) 2024-09-17T09:16:19.761 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T09:16:19.761 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-17T09:16:19.762 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-17T09:16:19.762 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-17T09:16:19.762 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.unordered_write: 0 2024-09-17T09:16:19.762 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T09:16:19.762 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T09:16:19.762 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T09:16:19.763 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T09:16:19.763 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.row_cache: None 2024-09-17T09:16:19.763 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.wal_filter: None 2024-09-17T09:16:19.763 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T09:16:19.763 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-17T09:16:19.763 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.preserve_deletes: 0 2024-09-17T09:16:19.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.two_write_queues: 0 2024-09-17T09:16:19.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.manual_wal_flush: 0 2024-09-17T09:16:19.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.atomic_flush: 0 2024-09-17T09:16:19.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T09:16:19.764 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T09:16:19.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T09:16:19.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.log_readahead_size: 0 2024-09-17T09:16:19.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-09-17T09:16:19.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.max_background_jobs: 2 2024-09-17T09:16:19.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.max_background_compactions: -1 2024-09-17T09:16:19.765 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T09:16:19.766 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T09:16:19.766 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T09:16:19.766 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.max_total_wal_size: 0 2024-09-17T09:16:19.766 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T09:16:19.766 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T09:16:19.767 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T09:16:19.767 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T09:16:19.767 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.max_open_files: -1 2024-09-17T09:16:19.767 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.bytes_per_sync: 0 2024-09-17T09:16:19.767 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T09:16:19.767 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T09:16:19.768 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-17T09:16:19.768 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Compression algorithms supported: 2024-09-17T09:16:19.768 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T09:16:19.768 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: kZSTD supported: 0 2024-09-17T09:16:19.768 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: kXpressCompression supported: 0 2024-09-17T09:16:19.768 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-17T09:16:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: kLZ4Compression supported: 1 2024-09-17T09:16:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: kBZip2Compression supported: 0 2024-09-17T09:16:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: kZlibCompression supported: 1 2024-09-17T09:16:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: kSnappyCompression supported: 1 2024-09-17T09:16:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T09:16:19.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.653+0000 7f5a96214700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-09-17T09:16:19.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-09-17T09:16:19.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-09-17T09:16:19.770 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T09:16:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.merge_operator: 2024-09-17T09:16:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compaction_filter: None 2024-09-17T09:16:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compaction_filter_factory: None 2024-09-17T09:16:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T09:16:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T09:16:19.771 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5624e23080d0) 2024-09-17T09:16:19.772 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-09-17T09:16:19.772 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T09:16:19.772 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T09:16:19.772 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-09-17T09:16:19.772 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: index_type: 0 2024-09-17T09:16:19.772 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-09-17T09:16:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-09-17T09:16:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-09-17T09:16:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-09-17T09:16:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: checksum: 1 2024-09-17T09:16:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-09-17T09:16:19.773 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_cache: 0x5624e233ed10 2024-09-17T09:16:19.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-09-17T09:16:19.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-09-17T09:16:19.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-09-17T09:16:19.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-09-17T09:16:19.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-09-17T09:16:19.774 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-09-17T09:16:19.775 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-09-17T09:16:19.775 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-09-17T09:16:19.775 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-09-17T09:16:19.775 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-09-17T09:16:19.775 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-09-17T09:16:19.776 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-09-17T09:16:19.776 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-09-17T09:16:19.776 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-09-17T09:16:19.776 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-09-17T09:16:19.776 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-17T09:16:19.776 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-09-17T09:16:19.777 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-09-17T09:16:19.777 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-09-17T09:16:19.777 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: format_version: 2 2024-09-17T09:16:19.777 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-09-17T09:16:19.777 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: block_align: 0 2024-09-17T09:16:19.778 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.778 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-17T09:16:19.778 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-17T09:16:19.778 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compression: NoCompression 2024-09-17T09:16:19.778 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-17T09:16:19.778 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-17T09:16:19.780 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T09:16:19.780 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.num_levels: 7 2024-09-17T09:16:19.780 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T09:16:19.780 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T09:16:19.780 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T09:16:19.780 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T09:16:19.780 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T09:16:19.780 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T09:16:19.780 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T09:16:19.781 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:16:19.781 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T09:16:19.781 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T09:16:19.781 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compression_opts.level: 32767 2024-09-17T09:16:19.781 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-17T09:16:19.781 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T09:16:19.781 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:16:19.781 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compression_opts.enabled: false 2024-09-17T09:16:19.781 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T09:16:19.781 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T09:16:19.781 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T09:16:19.781 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-17T09:16:19.781 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.arena_block_size: 4194304 2024-09-17T09:16:19.782 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T09:16:19.783 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T09:16:19.783 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-17T09:16:19.783 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-17T09:16:19.783 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T09:16:19.783 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T09:16:19.783 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T09:16:19.783 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T09:16:19.783 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T09:16:19.783 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T09:16:19.783 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T09:16:19.783 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T09:16:19.783 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T09:16:19.783 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T09:16:19.784 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.table_properties_collectors: 2024-09-17T09:16:19.784 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.inplace_update_support: 0 2024-09-17T09:16:19.784 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T09:16:19.784 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T09:16:19.784 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T09:16:19.784 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T09:16:19.784 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.bloom_locality: 0 2024-09-17T09:16:19.784 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.max_successive_merges: 0 2024-09-17T09:16:19.784 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T09:16:19.784 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-17T09:16:19.784 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.force_consistency_checks: 0 2024-09-17T09:16:19.784 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-17T09:16:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.ttl: 2592000 2024-09-17T09:16:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T09:16:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T09:16:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.654+0000 7f5a96214700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-09-17T09:16:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.655+0000 7f5a96214700 4 rocksdb: DB pointer 0x5624e2403800 2024-09-17T09:16:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.655+0000 7f5a7f0af700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-09-17T09:16:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.655+0000 7f5a7f0af700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-09-17T09:16:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-09-17T09:16:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:16:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T09:16:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:16:19.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:16:19.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T09:16:19.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-17T09:16:19.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:16:19.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-09-17T09:16:19.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:16:19.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:16:19.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T09:16:19.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T09:16:19.786 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-09-17T09:16:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:16:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:16:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:16:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T09:16:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:16:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:16:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:16:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:16:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:16:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:16:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:16:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:16:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-09-17T09:16:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:16:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:16:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T09:16:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T09:16:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-09-17T09:16:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:16:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:16:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:16:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T09:16:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:16:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:16:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:16:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:16:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:16:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:16:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:16:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:16:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: 2024-09-17T09:16:19.790 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.657+0000 7f5a96214700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-09-17T09:16:19.790 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.657+0000 7f5a96214700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-09-17T09:16:19.790 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph-mon: debug 2024-09-17T09:16:19.657+0000 7f5a96214700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-09-17T09:16:20.330 INFO:teuthology.orchestra.run.smithi029.stderr:create mon.a on 2024-09-17T09:16:20.331 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:16:20.338 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl disable ceph.target 2024-09-17T09:16:20.536 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl enable ceph.target 2024-09-17T09:16:20.557 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-17T09:16:20.703 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl start ceph.target 2024-09-17T09:16:20.711 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl enable ceph-69135c5a-74d5-11ef-bceb-c7b262605968.target 2024-09-17T09:16:20.719 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-69135c5a-74d5-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-69135c5a-74d5-11ef-bceb-c7b262605968.target. 2024-09-17T09:16:20.719 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-69135c5a-74d5-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-69135c5a-74d5-11ef-bceb-c7b262605968.target. 2024-09-17T09:16:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl start ceph-69135c5a-74d5-11ef-bceb-c7b262605968.target 2024-09-17T09:16:20.890 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl daemon-reload 2024-09-17T09:16:21.036 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.a 2024-09-17T09:16:21.066 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl reset-failed ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.a 2024-09-17T09:16:21.073 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Failed to reset failed state of unit ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-17T09:16:21.074 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl enable ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.a 2024-09-17T09:16:21.082 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/ceph-69135c5a-74d5-11ef-bceb-c7b262605968.target.wants/ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-69135c5a-74d5-11ef-bceb-c7b262605968@.service. 2024-09-17T09:16:21.229 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl start ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.a 2024-09-17T09:16:21.729 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled firewalld.service 2024-09-17T09:16:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: disabled 2024-09-17T09:16:21.739 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active firewalld.service 2024-09-17T09:16:21.745 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2024-09-17T09:16:21.746 INFO:teuthology.orchestra.run.smithi029.stderr:firewalld.service is not enabled 2024-09-17T09:16:21.746 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to enable service . firewalld.service is not available 2024-09-17T09:16:21.746 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-09-17T09:16:21.747 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mon to start... 2024-09-17T09:16:21.747 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mon... 2024-09-17T09:16:21.747 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status 2024-09-17T09:16:22.262 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: cluster: 2024-09-17T09:16:22.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: id: 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:16:22.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-09-17T09:16:22.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-09-17T09:16:22.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: services: 2024-09-17T09:16:22.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.599574s) 2024-09-17T09:16:22.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mgr: no daemons active 2024-09-17T09:16:22.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-09-17T09:16:22.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-09-17T09:16:22.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: data: 2024-09-17T09:16:22.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-09-17T09:16:22.263 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-09-17T09:16:22.264 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-09-17T09:16:22.264 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: pgs: 2024-09-17T09:16:22.264 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-09-17T09:16:23.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23324]: audit 2024-09-17T09:16:22.260421+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.29:0/1640887610' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-17T09:16:24.634 INFO:teuthology.orchestra.run.smithi029.stderr:mon is available 2024-09-17T09:16:24.634 INFO:teuthology.orchestra.run.smithi029.stderr:Assimilating anything we can from ceph.conf... 2024-09-17T09:16:24.635 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z -v /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a:/var/lib/ceph/mon/ceph-a:z quay.io/ceph/ceph:v16.2.0 config assimilate-conf -i /var/lib/ceph/mon/ceph-a/config 2024-09-17T09:16:25.136 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-09-17T09:16:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: [global] 2024-09-17T09:16:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: fsid = 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:16:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: log_to_journald = false 2024-09-17T09:16:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-09-17T09:16:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-09-17T09:16:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-09-17T09:16:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-09-17T09:16:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-09-17T09:16:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-09-17T09:16:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: [mgr] 2024-09-17T09:16:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-09-17T09:16:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-09-17T09:16:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-09-17T09:16:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: [mon] 2024-09-17T09:16:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T09:16:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T09:16:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-09-17T09:16:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: [osd] 2024-09-17T09:16:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-09-17T09:16:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-09-17T09:16:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-09-17T09:16:25.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23324]: audit 2024-09-17T09:16:25.131260+0000 mon.a (mon.0) 8 : audit [INF] from='client.? 172.21.15.29:0/2269952018' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-17T09:16:25.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23324]: audit 2024-09-17T09:16:25.133785+0000 mon.a (mon.0) 9 : audit [INF] from='client.? 172.21.15.29:0/2269952018' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-17T09:16:25.699 INFO:teuthology.orchestra.run.smithi029.stderr:Generating new minimal ceph.conf... 2024-09-17T09:16:25.699 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z -v /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a:/var/lib/ceph/mon/ceph-a:z quay.io/ceph/ceph:v16.2.0 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-a/config 2024-09-17T09:16:26.523 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23324]: audit 2024-09-17T09:16:26.201896+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.29:0/213855877' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:16:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:Restarting the monitor... 2024-09-17T09:16:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl restart ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.a 2024-09-17T09:16:27.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:26 smithi029 systemd[1]: Stopping Ceph mon.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:16:27.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23324]: debug 2024-09-17T09:16:26.938+0000 7fbbd69bf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-17T09:16:27.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23324]: debug 2024-09-17T09:16:26.938+0000 7fbbd69bf700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-17T09:16:27.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:26 smithi029 podman[23550]: 2024-09-17 09:16:26.979469663 +0000 UTC m=+0.170749670 container died 8133c18e6bca188a908418d579cff030d567d7004753d3bfad94bcafb8ec68e2 (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=HEAD, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557) 2024-09-17T09:16:27.573 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 podman[23550]: 2024-09-17 09:16:27.307012948 +0000 UTC m=+0.498292957 container cleanup 8133c18e6bca188a908418d579cff030d567d7004753d3bfad94bcafb8ec68e2 (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, RELEASE=HEAD, org.label-schema.build-date=20201204, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2024-09-17T09:16:27.573 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 podman[23550]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a 2024-09-17T09:16:27.573 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 podman[23562]: 2024-09-17 09:16:27.525514368 +0000 UTC m=+0.545579591 container remove 8133c18e6bca188a908418d579cff030d567d7004753d3bfad94bcafb8ec68e2 (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-17T09:16:27.573 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-17T09:16:27.574 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 systemd[1]: Stopped Ceph mon.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:16:27.574 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 systemd[1]: Starting Ceph mon.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:16:27.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 podman[23604]: 2024-09-17 09:16:27.710636725 +0000 UTC m=+0.106221073 container create f3476652dfdd603e0da45438ea41439f46b027ba5ee63edb952263d4c0be850f (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-16.2.0, RELEASE=HEAD, org.label-schema.build-date=20201204) 2024-09-17T09:16:27.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 podman[23604]: 2024-09-17 09:16:27.621900033 +0000 UTC m=+0.017484391 image pull 24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15 quay.io/ceph/ceph:v16.2.0 2024-09-17T09:16:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:Setting mon public_network to 172.21.0.0/20 2024-09-17T09:16:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config set mon public_network 172.21.0.0/20 2024-09-17T09:16:28.233 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 podman[23604]: 2024-09-17 09:16:27.914102106 +0000 UTC m=+0.309686457 container init f3476652dfdd603e0da45438ea41439f46b027ba5ee63edb952263d4c0be850f (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, ceph=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=-16.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-17T09:16:28.233 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 podman[23604]: 2024-09-17 09:16:27.917315866 +0000 UTC m=+0.312900208 container start f3476652dfdd603e0da45438ea41439f46b027ba5ee63edb952263d4c0be850f (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a, org.label-schema.build-date=20201204, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=-16.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-17T09:16:28.234 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.937+0000 7fcda4d14700 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-17T09:16:28.234 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.937+0000 7fcda4d14700 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-mon, pid 2 2024-09-17T09:16:28.234 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.937+0000 7fcda4d14700 0 pidfile_write: ignore empty --pid-file 2024-09-17T09:16:28.234 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.941+0000 7fcda4d14700 0 load: jerasure load: lrc load: isa 2024-09-17T09:16:28.234 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: RocksDB version: 6.8.1 2024-09-17T09:16:28.234 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.234 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-17T09:16:28.234 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Compile date Mar 30 2021 2024-09-17T09:16:28.234 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: DB SUMMARY 2024-09-17T09:16:28.234 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.234 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: CURRENT file: CURRENT 2024-09-17T09:16:28.234 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: IDENTITY file: IDENTITY 2024-09-17T09:16:28.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-09-17T09:16:28.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2024-09-17T09:16:28.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 83593 ; 2024-09-17T09:16:28.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.error_if_exists: 0 2024-09-17T09:16:28.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.create_if_missing: 0 2024-09-17T09:16:28.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.paranoid_checks: 1 2024-09-17T09:16:28.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.env: 0x55b06299d1c0 2024-09-17T09:16:28.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.fs: Posix File System 2024-09-17T09:16:28.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.info_log: 0x55b063cc2640 2024-09-17T09:16:28.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-17T09:16:28.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.statistics: (nil) 2024-09-17T09:16:28.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.use_fsync: 0 2024-09-17T09:16:28.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_log_file_size: 0 2024-09-17T09:16:28.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T09:16:28.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T09:16:28.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-17T09:16:28.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-17T09:16:28.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.allow_fallocate: 1 2024-09-17T09:16:28.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-17T09:16:28.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-17T09:16:28.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.use_direct_reads: 0 2024-09-17T09:16:28.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T09:16:28.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.create_missing_column_families: 0 2024-09-17T09:16:28.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.db_log_dir: 2024-09-17T09:16:28.238 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-17T09:16:28.238 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T09:16:28.238 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_subcompactions: 1 2024-09-17T09:16:28.238 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_background_flushes: -1 2024-09-17T09:16:28.238 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T09:16:28.238 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T09:16:28.238 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T09:16:28.238 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T09:16:28.238 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T09:16:28.238 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.advise_random_on_open: 1 2024-09-17T09:16:28.238 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-17T09:16:28.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.write_buffer_manager: 0x55b063d59e90 2024-09-17T09:16:28.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T09:16:28.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-17T09:16:28.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T09:16:28.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T09:16:28.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.rate_limiter: (nil) 2024-09-17T09:16:28.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T09:16:28.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-17T09:16:28.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-17T09:16:28.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-17T09:16:28.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.unordered_write: 0 2024-09-17T09:16:28.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T09:16:28.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T09:16:28.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T09:16:28.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T09:16:28.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.row_cache: None 2024-09-17T09:16:28.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.wal_filter: None 2024-09-17T09:16:28.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T09:16:28.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-17T09:16:28.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.preserve_deletes: 0 2024-09-17T09:16:28.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.two_write_queues: 0 2024-09-17T09:16:28.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.manual_wal_flush: 0 2024-09-17T09:16:28.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.atomic_flush: 0 2024-09-17T09:16:28.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T09:16:28.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T09:16:28.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T09:16:28.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.log_readahead_size: 0 2024-09-17T09:16:28.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-09-17T09:16:28.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_background_jobs: 2 2024-09-17T09:16:28.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_background_compactions: -1 2024-09-17T09:16:28.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T09:16:28.241 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T09:16:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T09:16:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_total_wal_size: 0 2024-09-17T09:16:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T09:16:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T09:16:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T09:16:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T09:16:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_open_files: -1 2024-09-17T09:16:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.bytes_per_sync: 0 2024-09-17T09:16:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T09:16:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T09:16:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-17T09:16:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Compression algorithms supported: 2024-09-17T09:16:28.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T09:16:28.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: kZSTD supported: 0 2024-09-17T09:16:28.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: kXpressCompression supported: 0 2024-09-17T09:16:28.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-17T09:16:28.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: kLZ4Compression supported: 1 2024-09-17T09:16:28.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: kBZip2Compression supported: 0 2024-09-17T09:16:28.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: kZlibCompression supported: 1 2024-09-17T09:16:28.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: kSnappyCompression supported: 1 2024-09-17T09:16:28.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T09:16:28.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2024-09-17T09:16:28.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-09-17T09:16:28.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T09:16:28.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.merge_operator: 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compaction_filter: None 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compaction_filter_factory: None 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b063c680d8) 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cache_index_and_filter_blocks: 1 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: pin_top_level_index_and_filter: 1 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: index_type: 0 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: data_block_index_type: 0 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: index_shortening: 1 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: data_block_hash_table_util_ratio: 0.750000 2024-09-17T09:16:28.246 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: hash_index_allow_collision: 1 2024-09-17T09:16:28.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: checksum: 1 2024-09-17T09:16:28.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: no_block_cache: 0 2024-09-17T09:16:28.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: block_cache: 0x55b063c9ef10 2024-09-17T09:16:28.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: block_cache_name: BinnedLRUCache 2024-09-17T09:16:28.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: block_cache_options: 2024-09-17T09:16:28.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: capacity : 536870912 2024-09-17T09:16:28.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: num_shard_bits : 4 2024-09-17T09:16:28.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: strict_capacity_limit : 0 2024-09-17T09:16:28.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: high_pri_pool_ratio: 0.000 2024-09-17T09:16:28.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: block_cache_compressed: (nil) 2024-09-17T09:16:28.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: persistent_cache: (nil) 2024-09-17T09:16:28.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: block_size: 4096 2024-09-17T09:16:28.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: block_size_deviation: 10 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: block_restart_interval: 16 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: index_block_restart_interval: 1 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: metadata_block_size: 4096 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: partition_filters: 0 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: use_delta_encoding: 1 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: whole_key_filtering: 1 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: verify_compression: 0 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: read_amp_bytes_per_bit: 0 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: format_version: 2 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: enable_index_compression: 1 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: block_align: 0 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-17T09:16:28.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-17T09:16:28.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compression: NoCompression 2024-09-17T09:16:28.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-17T09:16:28.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-17T09:16:28.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T09:16:28.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.num_levels: 7 2024-09-17T09:16:28.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T09:16:28.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T09:16:28.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T09:16:28.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T09:16:28.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T09:16:28.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T09:16:28.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T09:16:28.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:16:28.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T09:16:28.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T09:16:28.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compression_opts.level: 32767 2024-09-17T09:16:28.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-17T09:16:28.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T09:16:28.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:16:28.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compression_opts.enabled: false 2024-09-17T09:16:28.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T09:16:28.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T09:16:28.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T09:16:28.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-17T09:16:28.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T09:16:28.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T09:16:28.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T09:16:28.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T09:16:28.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T09:16:28.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T09:16:28.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T09:16:28.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T09:16:28.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T09:16:28.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T09:16:28.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T09:16:28.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T09:16:28.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T09:16:28.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.arena_block_size: 4194304 2024-09-17T09:16:28.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T09:16:28.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T09:16:28.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-17T09:16:28.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-17T09:16:28.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T09:16:28.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T09:16:28.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T09:16:28.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T09:16:28.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T09:16:28.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T09:16:28.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T09:16:28.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T09:16:28.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T09:16:28.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T09:16:28.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.table_properties_collectors: 2024-09-17T09:16:28.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.inplace_update_support: 0 2024-09-17T09:16:28.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T09:16:28.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T09:16:28.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T09:16:28.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T09:16:28.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.bloom_locality: 0 2024-09-17T09:16:28.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.max_successive_merges: 0 2024-09-17T09:16:28.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T09:16:28.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-17T09:16:28.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.force_consistency_checks: 0 2024-09-17T09:16:28.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-17T09:16:28.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.ttl: 2592000 2024-09-17T09:16:28.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.942+0000 7fcda4d14700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T09:16:28.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.943+0000 7fcda4d14700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T09:16:28.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.943+0000 7fcda4d14700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-09-17T09:16:28.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.943+0000 7fcda4d14700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726564587943885, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-09-17T09:16:28.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.943+0000 7fcda4d14700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-09-17T09:16:28.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.943+0000 7fcda4d14700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-17T09:16:28.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.944+0000 7fcda4d14700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726564587944792, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 80317, "table_properties": {"data_size": 78575, "index_size": 347, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 8995, "raw_average_key_size": 46, "raw_value_size": 73321, "raw_average_value_size": 376, "num_data_blocks": 11, "num_entries": 195, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726564587, "oldest_key_time": 3, "file_creation_time": 0}} 2024-09-17T09:16:28.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.944+0000 7fcda4d14700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-09-17T09:16:28.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.944+0000 7fcda4d14700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726564587945191, "job": 1, "event": "recovery_finished"} 2024-09-17T09:16:28.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.945+0000 7fcda4d14700 4 rocksdb: DB pointer 0x55b063d63800 2024-09-17T09:16:28.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.945+0000 7fcd8bb2e700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-09-17T09:16:28.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.945+0000 7fcd8bb2e700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-09-17T09:16:28.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: ** DB Stats ** 2024-09-17T09:16:28.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:16:28.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T09:16:28.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:16:28.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:16:28.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T09:16:28.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-17T09:16:28.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:16:28.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: ** Compaction Stats [default] ** 2024-09-17T09:16:28.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:16:28.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:16:28.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: L0 2/0 80.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.1 0.00 0.00 1 0.001 0 0 2024-09-17T09:16:28.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Sum 2/0 80.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.1 0.00 0.00 1 0.001 0 0 2024-09-17T09:16:28.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.1 0.00 0.00 1 0.001 0 0 2024-09-17T09:16:28.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: ** Compaction Stats [default] ** 2024-09-17T09:16:28.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:16:28.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:16:28.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 128.1 0.00 0.00 1 0.001 0 0 2024-09-17T09:16:28.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:16:28.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T09:16:28.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:16:28.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:16:28.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:16:28.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:16:28.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Cumulative compaction: 0.00 GB write, 25.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:16:28.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Interval compaction: 0.00 GB write, 25.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:16:28.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:16:28.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:16:28.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: ** Compaction Stats [default] ** 2024-09-17T09:16:28.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:16:28.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:16:28.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: L0 2/0 80.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.1 0.00 0.00 1 0.001 0 0 2024-09-17T09:16:28.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Sum 2/0 80.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.1 0.00 0.00 1 0.001 0 0 2024-09-17T09:16:28.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T09:16:28.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: ** Compaction Stats [default] ** 2024-09-17T09:16:28.260 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:16:28.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:16:28.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 128.1 0.00 0.00 1 0.001 0 0 2024-09-17T09:16:28.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:16:28.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T09:16:28.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:16:28.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:16:28.261 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:16:28.262 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:16:28.262 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Cumulative compaction: 0.00 GB write, 25.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:16:28.262 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:16:28.262 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:16:28.262 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.262 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:16:28.262 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.262 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.945+0000 7fcda4d14700 0 starting mon.a rank 0 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:16:28.262 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.946+0000 7fcda4d14700 1 mon.a@-1(???) e1 preinit fsid 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.947+0000 7fcda4d14700 0 mon.a@-1(???).mds e1 new map 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.947+0000 7fcda4d14700 0 mon.a@-1(???).mds e1 print_map 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: e1 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: legacy client fscid: -1 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: No filesystems configured 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.947+0000 7fcda4d14700 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.947+0000 7fcda4d14700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.947+0000 7fcda4d14700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.947+0000 7fcda4d14700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:16:28.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:26.201896+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.29:0/213855877' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:16:28.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.947+0000 7fcda4d14700 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-17T09:16:28.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.948+0000 7fcda4d14700 20 expand_channel_meta expand map: {default=false} 2024-09-17T09:16:28.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.948+0000 7fcda4d14700 20 expand_channel_meta from 'false' to 'false' 2024-09-17T09:16:28.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.948+0000 7fcda4d14700 20 expand_channel_meta expanded map: {default=false} 2024-09-17T09:16:28.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.948+0000 7fcda4d14700 20 expand_channel_meta expand map: {default=info} 2024-09-17T09:16:28.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.948+0000 7fcda4d14700 20 expand_channel_meta from 'info' to 'info' 2024-09-17T09:16:28.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.948+0000 7fcda4d14700 20 expand_channel_meta expanded map: {default=info} 2024-09-17T09:16:28.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.948+0000 7fcda4d14700 20 expand_channel_meta expand map: {default=daemon} 2024-09-17T09:16:28.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.948+0000 7fcda4d14700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-09-17T09:16:28.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.948+0000 7fcda4d14700 20 expand_channel_meta expanded map: {default=daemon} 2024-09-17T09:16:28.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.948+0000 7fcda4d14700 20 expand_channel_meta expand map: {default=debug} 2024-09-17T09:16:28.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.948+0000 7fcda4d14700 20 expand_channel_meta from 'debug' to 'debug' 2024-09-17T09:16:28.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:16:27.948+0000 7fcda4d14700 20 expand_channel_meta expanded map: {default=debug} 2024-09-17T09:16:28.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:27.951283+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T09:16:28.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:27.951326+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} 2024-09-17T09:16:28.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:27.951357+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2024-09-17T09:16:28.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:27.951379+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-09-17T09:16:28.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:27.951684+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-09-17T09:16:28.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:28 smithi029 bash[23604]: f3476652dfdd603e0da45438ea41439f46b027ba5ee63edb952263d4c0be850f 2024-09-17T09:16:28.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:28 smithi029 systemd[1]: Started Ceph mon.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:16:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:Wrote config to /etc/ceph/ceph.conf 2024-09-17T09:16:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:16:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:Creating mgr... 2024-09-17T09:16:29.093 INFO:teuthology.orchestra.run.smithi029.stderr:Verifying port 9283 ... 2024-09-17T09:16:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl daemon-reload 2024-09-17T09:16:29.250 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.y 2024-09-17T09:16:29.265 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl reset-failed ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.y 2024-09-17T09:16:29.271 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Failed to reset failed state of unit ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-17T09:16:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl enable ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.y 2024-09-17T09:16:29.279 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: Created symlink /etc/systemd/system/ceph-69135c5a-74d5-11ef-bceb-c7b262605968.target.wants/ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-69135c5a-74d5-11ef-bceb-c7b262605968@.service. 2024-09-17T09:16:29.427 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl start ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.y 2024-09-17T09:16:29.446 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:29 smithi029 systemd[1]: Starting Ceph mgr.y for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:16:29.705 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:28.499365+0000 mon.a (mon.0) 6 : audit [INF] from='client.? 172.21.15.29:0/1028590453' entity='client.admin' 2024-09-17T09:16:29.705 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:29 smithi029 podman[23816]: 2024-09-17 09:16:29.606806877 +0000 UTC m=+0.103358829 container create 9d130c67b36dadbd4e03fdaf106266cf5fd868b8f96d8dfd5ce819a4552171a2 (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20201204) 2024-09-17T09:16:29.706 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:29 smithi029 podman[23816]: 2024-09-17 09:16:29.519603181 +0000 UTC m=+0.016155132 image pull 24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15 quay.io/ceph/ceph:v16.2.0 2024-09-17T09:16:29.909 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled firewalld.service 2024-09-17T09:16:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: disabled 2024-09-17T09:16:29.919 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active firewalld.service 2024-09-17T09:16:29.927 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2024-09-17T09:16:29.928 INFO:teuthology.orchestra.run.smithi029.stderr:firewalld.service is not enabled 2024-09-17T09:16:29.928 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to enable service . firewalld.service is not available 2024-09-17T09:16:29.928 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-09-17T09:16:29.929 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled firewalld.service 2024-09-17T09:16:29.937 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: disabled 2024-09-17T09:16:29.938 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active firewalld.service 2024-09-17T09:16:29.945 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2024-09-17T09:16:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:firewalld.service is not enabled 2024-09-17T09:16:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-09-17T09:16:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mgr to start... 2024-09-17T09:16:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mgr... 2024-09-17T09:16:29.946 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status -f json-pretty 2024-09-17T09:16:30.047 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:29 smithi029 podman[23816]: 2024-09-17 09:16:29.811832048 +0000 UTC m=+0.308383999 container init 9d130c67b36dadbd4e03fdaf106266cf5fd868b8f96d8dfd5ce819a4552171a2 (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20201204, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD) 2024-09-17T09:16:30.048 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:29 smithi029 podman[23816]: 2024-09-17 09:16:29.815022146 +0000 UTC m=+0.311574097 container start 9d130c67b36dadbd4e03fdaf106266cf5fd868b8f96d8dfd5ce819a4552171a2 (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Dimitri Savineau ) 2024-09-17T09:16:30.048 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:29 smithi029 bash[23816]: 9d130c67b36dadbd4e03fdaf106266cf5fd868b8f96d8dfd5ce819a4552171a2 2024-09-17T09:16:30.048 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:29 smithi029 systemd[1]: Started Ceph mgr.y for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:16:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-09-17T09:16:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-09-17T09:16:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsid": "69135c5a-74d5-11ef-bceb-c7b262605968", 2024-09-17T09:16:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "health": { 2024-09-17T09:16:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-17T09:16:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "checks": {}, 2024-09-17T09:16:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mutes": [] 2024-09-17T09:16:30.455 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-17T09:16:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum": [ 2024-09-17T09:16:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 0 2024-09-17T09:16:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-09-17T09:16:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-17T09:16:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "a" 2024-09-17T09:16:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-09-17T09:16:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-09-17T09:16:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "monmap": { 2024-09-17T09:16:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T09:16:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-09-17T09:16:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-17T09:16:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osdmap": { 2024-09-17T09:16:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T09:16:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-17T09:16:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-17T09:16:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-17T09:16:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-17T09:16:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-17T09:16:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-17T09:16:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgmap": { 2024-09-17T09:16:30.457 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsmap": { 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:30.458 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-17T09:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": false, 2024-09-17T09:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-17T09:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modules": [ 2024-09-17T09:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "iostat", 2024-09-17T09:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "restful" 2024-09-17T09:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-09-17T09:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-09-17T09:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "servicemap": { 2024-09-17T09:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T09:16:30.459 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modified": "2024-09-17T09:16:21.661676+0000", 2024-09-17T09:16:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-09-17T09:16:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-17T09:16:30.460 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-09-17T09:16:30.742 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:30.447104+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.29:0/3901578062' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T09:16:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:mgr not available, waiting (1/15)... 2024-09-17T09:16:33.032 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status -f json-pretty 2024-09-17T09:16:33.546 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-09-17T09:16:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-09-17T09:16:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsid": "69135c5a-74d5-11ef-bceb-c7b262605968", 2024-09-17T09:16:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "health": { 2024-09-17T09:16:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-17T09:16:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "checks": {}, 2024-09-17T09:16:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mutes": [] 2024-09-17T09:16:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-17T09:16:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum": [ 2024-09-17T09:16:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 0 2024-09-17T09:16:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "a" 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "monmap": { 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osdmap": { 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-17T09:16:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-17T09:16:33.552 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-17T09:16:33.552 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-17T09:16:33.552 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-17T09:16:33.552 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgmap": { 2024-09-17T09:16:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-17T09:16:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-17T09:16:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-17T09:16:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-17T09:16:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-17T09:16:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-17T09:16:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-17T09:16:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-17T09:16:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsmap": { 2024-09-17T09:16:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T09:16:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-17T09:16:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-17T09:16:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-17T09:16:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": false, 2024-09-17T09:16:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-17T09:16:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modules": [ 2024-09-17T09:16:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "iostat", 2024-09-17T09:16:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "restful" 2024-09-17T09:16:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-09-17T09:16:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-09-17T09:16:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "servicemap": { 2024-09-17T09:16:33.555 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T09:16:33.555 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modified": "2024-09-17T09:16:21.661676+0000", 2024-09-17T09:16:33.555 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-09-17T09:16:33.555 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:33.555 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-17T09:16:33.555 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-09-17T09:16:34.170 INFO:teuthology.orchestra.run.smithi029.stderr:mgr not available, waiting (2/15)... 2024-09-17T09:16:34.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:33.543515+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? 172.21.15.29:0/3452560416' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T09:16:35.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:34.919945+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2024-09-17T09:16:35.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:34.922633+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00278023s) 2024-09-17T09:16:35.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:34.923469+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.29:0/2260982867' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:16:35.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:34.923692+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.29:0/2260982867' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:16:35.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:34.923873+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.29:0/2260982867' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:16:35.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:34.924030+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.29:0/2260982867' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:16:35.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:34.924238+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.29:0/2260982867' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:16:35.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:34.925938+0000 mon.a (mon.0) 16 : cluster [INF] Manager daemon y is now available 2024-09-17T09:16:35.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:34.933577+0000 mon.a (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.29:0/2260982867' entity='mgr.y' 2024-09-17T09:16:35.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:34.937316+0000 mon.a (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.29:0/2260982867' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:16:35.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:34.938453+0000 mon.a (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.29:0/2260982867' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:16:35.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:34.942804+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.29:0/2260982867' entity='mgr.y' 2024-09-17T09:16:35.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:34.944517+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.29:0/2260982867' entity='mgr.y' 2024-09-17T09:16:36.172 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 status -f json-pretty 2024-09-17T09:16:36.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 2024-09-17T09:16:36.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-09-17T09:16:36.810 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsid": "69135c5a-74d5-11ef-bceb-c7b262605968", 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "health": { 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "checks": {}, 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mutes": [] 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum": [ 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 0 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "a" 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-09-17T09:16:36.811 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "monmap": { 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osdmap": { 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:36.812 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgmap": { 2024-09-17T09:16:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-17T09:16:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-17T09:16:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-17T09:16:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-17T09:16:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-17T09:16:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-17T09:16:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-17T09:16:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-17T09:16:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "fsmap": { 2024-09-17T09:16:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T09:16:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-17T09:16:36.813 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": true, 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modules": [ 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "iostat", 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "restful" 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ], 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "servicemap": { 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "modified": "2024-09-17T09:16:21.661676+0000", 2024-09-17T09:16:36.814 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "services": {} 2024-09-17T09:16:36.815 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: }, 2024-09-17T09:16:36.815 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-17T09:16:36.815 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-09-17T09:16:37.288 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:35.925870+0000 mon.a (mon.0) 22 : cluster [DBG] mgrmap e3: y(active, since 1.00602s) 2024-09-17T09:16:37.289 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:36.803774+0000 mon.a (mon.0) 23 : audit [DBG] from='client.? 172.21.15.29:0/3621095710' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T09:16:37.383 INFO:teuthology.orchestra.run.smithi029.stderr:mgr is available 2024-09-17T09:16:37.383 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config-key set mgr/cephadm/ssh_user root 2024-09-17T09:16:38.037 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: set mgr/cephadm/ssh_user 2024-09-17T09:16:38.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:36.926207+0000 mon.a (mon.0) 24 : cluster [DBG] mgrmap e4: y(active, since 2s) 2024-09-17T09:16:38.547 INFO:teuthology.orchestra.run.smithi029.stderr:Enabling cephadm module... 2024-09-17T09:16:38.547 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr module enable cephadm 2024-09-17T09:16:39.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:38.032405+0000 mon.a (mon.0) 25 : audit [INF] from='client.? 172.21.15.29:0/347516436' entity='client.admin' 2024-09-17T09:16:40.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:39.143891+0000 mon.a (mon.0) 26 : audit [INF] from='client.? 172.21.15.29:0/1842198437' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-17T09:16:40.298 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: ignoring --setuser ceph since I am not root 2024-09-17T09:16:40.298 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: ignoring --setgroup ceph since I am not root 2024-09-17T09:16:40.627 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr stat 2024-09-17T09:16:41.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:40.036744+0000 mon.a (mon.0) 27 : audit [INF] from='client.? 172.21.15.29:0/1842198437' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-17T09:16:41.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:40.036797+0000 mon.a (mon.0) 28 : cluster [DBG] mgrmap e5: y(active, since 5s) 2024-09-17T09:16:41.237 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-09-17T09:16:41.237 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 5, 2024-09-17T09:16:41.237 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": true, 2024-09-17T09:16:41.237 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "active_name": "y", 2024-09-17T09:16:41.238 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standby": 0 2024-09-17T09:16:41.238 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-09-17T09:16:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for the mgr to restart... 2024-09-17T09:16:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mgr epoch 5... 2024-09-17T09:16:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 tell mgr mgr_status 2024-09-17T09:16:42.219 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:41.235759+0000 mon.a (mon.0) 29 : audit [DBG] from='client.? 172.21.15.29:0/661083935' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-17T09:16:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-09-17T09:16:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-09-17T09:16:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "initialized": true 2024-09-17T09:16:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-09-17T09:16:46.433 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:45.142188+0000 mon.a (mon.0) 30 : cluster [INF] Active manager daemon y restarted 2024-09-17T09:16:46.437 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:45.142677+0000 mon.a (mon.0) 31 : cluster [INF] Activating manager daemon y 2024-09-17T09:16:46.437 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:45.144320+0000 mon.a (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-09-17T09:16:46.437 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:45.195155+0000 mon.a (mon.0) 33 : cluster [DBG] mgrmap e6: y(active, starting, since 0.0525944s) 2024-09-17T09:16:46.437 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.195869+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:16:46.437 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.196467+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:16:46.437 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.197048+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:16:46.437 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.197209+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:16:46.437 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.197353+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:16:46.437 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:45.199161+0000 mon.a (mon.0) 39 : cluster [INF] Manager daemon y is now available 2024-09-17T09:16:46.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.212133+0000 mon.a (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' 2024-09-17T09:16:46.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.214763+0000 mon.a (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' 2024-09-17T09:16:46.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.217114+0000 mon.a (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' 2024-09-17T09:16:46.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.312973+0000 mon.a (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' 2024-09-17T09:16:46.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.313656+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:16:46.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.315063+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:16:46.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.316370+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:16:46.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.327483+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:16:46.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:45.328799+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:16:46.748 INFO:teuthology.orchestra.run.smithi029.stderr:mgr epoch 5 is available 2024-09-17T09:16:46.748 INFO:teuthology.orchestra.run.smithi029.stderr:Setting orchestrator backend to cephadm... 2024-09-17T09:16:46.749 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch set backend cephadm 2024-09-17T09:16:47.375 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:46.197298+0000 mgr.y (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-17T09:16:47.375 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:46.197976+0000 mon.a (mon.0) 49 : cluster [DBG] mgrmap e7: y(active, since 1.05541s) 2024-09-17T09:16:47.375 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:46.204210+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-17T09:16:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:Generating ssh key... 2024-09-17T09:16:47.909 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 cephadm generate-key 2024-09-17T09:16:48.288 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:47.199527+0000 mon.a (mon.0) 50 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-09-17T09:16:48.288 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:47.358073+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:16:48.288 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:47.361857+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' 2024-09-17T09:16:48.288 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:47.364856+0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:16:48.944 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: Generating public/private rsa key pair. 2024-09-17T09:16:48.944 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: Your identification has been saved in /tmp/tmp5l_ohh4d/key. 2024-09-17T09:16:48.944 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: Your public key has been saved in /tmp/tmp5l_ohh4d/key.pub. 2024-09-17T09:16:48.944 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: The key fingerprint is: 2024-09-17T09:16:48.944 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: SHA256:n/YSaQ5BL4deKioqFJUI69vGz0rW5Owfs5HVA5d7t5c ceph-69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:16:48.944 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: The key's randomart image is: 2024-09-17T09:16:48.944 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: +---[RSA 3072]----+ 2024-09-17T09:16:48.944 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: |o . . | 2024-09-17T09:16:48.944 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: | o o . . | 2024-09-17T09:16:48.945 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: |. . . + o | 2024-09-17T09:16:48.945 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: |.. + B . | 2024-09-17T09:16:48.945 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: | .. . .SB = . . | 2024-09-17T09:16:48.945 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: | .+ = . *.+.o . o| 2024-09-17T09:16:48.945 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: |.. * = * ++. E.| 2024-09-17T09:16:48.945 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: |. = = =.o. .| 2024-09-17T09:16:48.945 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: |o. o.+.o .. | 2024-09-17T09:16:48.945 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: +----[SHA256]-----+ 2024-09-17T09:16:49.191 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 cephadm get-pub-key 2024-09-17T09:16:49.747 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8S6QmDaZ+moU3SA7wBy94ACGcgcxEJJZg6cooVj0JRa8FLUrbdWN/beGdj/Gc41m+svwHNyLH7RAIr72rwQtg4tUncLmGlp22+JeBOApdhb4B0owbyzKIYTJWbawkbVDPegbO0nmGt5srT8JTQfhHRceIN99HwIy+xxtv0/BOBhVeyho7NvaC/mQNve940U9PrMoY9b+nqG6ThTf6gyYHLzoOxtUqYflwtvcQfDVHeShAs6taIxxL9mL9+KuqYilBCKFJW32lp0vhrABqY1z8SAR0d8IlM6LSHazYWQc9ZIOmY8/TeOvrwh2Kc4c6AgIn8LkSudGlmQ9jw3J9p9r17HEthI7eNFCh3mUDe4XpsQtw0LXD4w39pPPnBUntqE/r4aTCfUk7bt+NlYX7AtK4Jd5nCaTTbkcKi98QKOOlsGNz0JvIqrYAmuE3U+w8ufPjoaKkpYbaxve+6bzI7RZNycLEzN8H7yb8RY9dRykc3A2rpJwtLmfcfIbcAbkc40c= ceph-69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:16:50.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:16:48.565251+0000 mgr.y (mgr.14116) 5 : cephadm [INF] Generating ssh key... 2024-09-17T09:16:50.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:48.673556+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' 2024-09-17T09:16:50.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:48.675167+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' 2024-09-17T09:16:50.287 INFO:teuthology.orchestra.run.smithi029.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-17T09:16:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:Adding key to root@localhost authorized_keys... 2024-09-17T09:16:50.292 INFO:teuthology.orchestra.run.smithi029.stderr:Adding host smithi029... 2024-09-17T09:16:50.292 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch host add smithi029 2024-09-17T09:16:50.929 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:49.746425+0000 mgr.y (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:16:51.297 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: Warning: Permanently added 'smithi029,172.21.15.29' (ECDSA) to the list of known hosts. 2024-09-17T09:16:51.535 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Added host 'smithi029' 2024-09-17T09:16:51.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:50.892580+0000 mgr.y (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:16:51.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:51.532538+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' 2024-09-17T09:16:51.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:51.533125+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:16:52.061 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying unmanaged mon service... 2024-09-17T09:16:52.061 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch apply mon --unmanaged 2024-09-17T09:16:52.687 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled mon update... 2024-09-17T09:16:52.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:16:51.532862+0000 mgr.y (mgr.14116) 8 : cephadm [INF] Added host smithi029 2024-09-17T09:16:53.165 INFO:teuthology.orchestra.run.smithi029.stderr:Deploying unmanaged mgr service... 2024-09-17T09:16:53.165 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 orch apply mgr --unmanaged 2024-09-17T09:16:53.769 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-09-17T09:16:53.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:52.682372+0000 mgr.y (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:16:53.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:16:52.683119+0000 mgr.y (mgr.14116) 10 : cephadm [INF] Saving service mon spec with placement count:5 2024-09-17T09:16:53.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:52.686200+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' 2024-09-17T09:16:55.021 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:53.765777+0000 mgr.y (mgr.14116) 11 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:16:55.021 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:16:53.766248+0000 mgr.y (mgr.14116) 12 : cephadm [INF] Saving service mgr spec with placement count:2 2024-09-17T09:16:55.021 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:53.768276+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' 2024-09-17T09:16:55.418 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config set mgr mgr/cephadm/container_init True --force 2024-09-17T09:16:56.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:55.049064+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' 2024-09-17T09:16:56.133 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:55.261209+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' 2024-09-17T09:16:56.735 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config set mgr mgr/dashboard/ssl_server_port 8443 2024-09-17T09:16:57.148 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:56.066459+0000 mon.a (mon.0) 61 : audit [INF] from='client.? 172.21.15.29:0/1621944067' entity='client.admin' 2024-09-17T09:16:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:Enabling the dashboard module... 2024-09-17T09:16:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr module enable dashboard 2024-09-17T09:16:58.465 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:57.321055+0000 mon.a (mon.0) 62 : audit [INF] from='client.? 172.21.15.29:0/1643504331' entity='client.admin' 2024-09-17T09:16:58.465 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:57.639260+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.14116 172.21.15.29:0/2625041176' entity='mgr.y' 2024-09-17T09:16:59.596 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:16:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:58.781277+0000 mon.a (mon.0) 64 : audit [INF] from='client.? 172.21.15.29:0/216446956' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-17T09:16:59.596 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: ignoring --setuser ceph since I am not root 2024-09-17T09:16:59.596 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:16:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: ignoring --setgroup ceph since I am not root 2024-09-17T09:17:00.273 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 mgr stat 2024-09-17T09:17:00.525 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:16:59.324845+0000 mon.a (mon.0) 65 : audit [INF] from='client.? 172.21.15.29:0/216446956' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-17T09:17:00.525 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:16:59.324896+0000 mon.a (mon.0) 66 : cluster [DBG] mgrmap e9: y(active, since 14s) 2024-09-17T09:17:00.955 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-09-17T09:17:00.955 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "epoch": 9, 2024-09-17T09:17:00.955 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "available": true, 2024-09-17T09:17:00.955 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "active_name": "y", 2024-09-17T09:17:00.955 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "num_standby": 0 2024-09-17T09:17:00.955 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-09-17T09:17:01.495 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:00.953752+0000 mon.a (mon.0) 67 : audit [DBG] from='client.? 172.21.15.29:0/2709848127' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-17T09:17:01.497 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for the mgr to restart... 2024-09-17T09:17:01.498 INFO:teuthology.orchestra.run.smithi029.stderr:Waiting for mgr epoch 9... 2024-09-17T09:17:01.498 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 tell mgr mgr_status 2024-09-17T09:17:05.047 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:17:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: Warning: Permanently added 'smithi029,172.21.15.29' (ECDSA) to the list of known hosts. 2024-09-17T09:17:05.543 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: { 2024-09-17T09:17:05.544 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-09-17T09:17:05.544 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: "initialized": true 2024-09-17T09:17:05.544 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: } 2024-09-17T09:17:05.732 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:04.481002+0000 mon.a (mon.0) 68 : cluster [INF] Active manager daemon y restarted 2024-09-17T09:17:05.732 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:04.481444+0000 mon.a (mon.0) 69 : cluster [INF] Activating manager daemon y 2024-09-17T09:17:05.732 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:04.482968+0000 mon.a (mon.0) 70 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-09-17T09:17:05.732 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:04.533887+0000 mon.a (mon.0) 71 : cluster [DBG] mgrmap e10: y(active, starting, since 0.0525348s) 2024-09-17T09:17:05.732 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:04.534978+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:17:05.732 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:04.535560+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:17:05.732 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:04.536339+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:17:05.732 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:04.536570+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:17:05.732 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:04.536788+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:17:05.733 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:04.538676+0000 mon.a (mon.0) 77 : cluster [INF] Manager daemon y is now available 2024-09-17T09:17:05.733 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:04.644450+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:05.733 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:04.647030+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:05.733 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:04.666277+0000 mon.a (mon.0) 80 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:17:05.733 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:04.667864+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:17:06.337 INFO:teuthology.orchestra.run.smithi029.stderr:mgr epoch 9 is available 2024-09-17T09:17:06.337 INFO:teuthology.orchestra.run.smithi029.stderr:Generating a dashboard self-signed certificate... 2024-09-17T09:17:06.337 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 dashboard create-self-signed-cert 2024-09-17T09:17:06.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:05.535308+0000 mgr.y (mgr.14144) 1 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-17T09:17:06.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:05.536068+0000 mon.a (mon.0) 82 : cluster [DBG] mgrmap e11: y(active, since 1.05471s) 2024-09-17T09:17:06.591 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:05.541717+0000 mgr.y (mgr.14144) 2 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-17T09:17:06.591 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:05.975786+0000 mon.a (mon.0) 83 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:07.120 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: Self-signed certificate created 2024-09-17T09:17:08.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:06.977113+0000 mon.a (mon.0) 84 : cluster [DBG] mgrmap e12: y(active, since 2s) 2024-09-17T09:17:08.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:07.003003+0000 mgr.y (mgr.14144) 3 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:17:08.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:07.116543+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:08.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:07.118702+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:08.544 INFO:teuthology.orchestra.run.smithi029.stderr:Creating initial admin user... 2024-09-17T09:17:08.544 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpc050vgc3:/tmp/dashboard.pw:z quay.io/ceph/ceph:v16.2.0 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-09-17T09:17:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$fdWSnJxMvo434CoJKATYcOPV9s75n0fqIBCrTK5EWh09cpsdqn1e.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726564629, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-17T09:17:09.794 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:09.186544+0000 mgr.y (mgr.14144) 4 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:17:10.059 INFO:teuthology.orchestra.run.smithi029.stderr:Fetching dashboard port number... 2024-09-17T09:17:10.059 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16.2.0 -e NODE_NAME=smithi029 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968:/var/log/ceph:z -v /tmp/ceph-tmp6dhc4k02:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp1ezilb1p:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v16.2.0 config get mgr mgr/dashboard/ssl_server_port 2024-09-17T09:17:10.641 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:09.430765+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:10.641 INFO:teuthology.orchestra.run.smithi029.stderr:/usr/bin/ceph: 8443 2024-09-17T09:17:11.223 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-enabled firewalld.service 2024-09-17T09:17:11.232 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: disabled 2024-09-17T09:17:11.233 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: systemctl is-active firewalld.service 2024-09-17T09:17:11.240 INFO:teuthology.orchestra.run.smithi029.stderr:systemctl: inactive 2024-09-17T09:17:11.240 INFO:teuthology.orchestra.run.smithi029.stderr:firewalld.service is not enabled 2024-09-17T09:17:11.241 INFO:teuthology.orchestra.run.smithi029.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-17T09:17:11.244 INFO:teuthology.orchestra.run.smithi029.stderr:Ceph Dashboard is now available at: 2024-09-17T09:17:11.244 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-09-17T09:17:11.244 INFO:teuthology.orchestra.run.smithi029.stderr: URL: https://smithi029.front.sepia.ceph.com:8443/ 2024-09-17T09:17:11.244 INFO:teuthology.orchestra.run.smithi029.stderr: User: admin 2024-09-17T09:17:11.244 INFO:teuthology.orchestra.run.smithi029.stderr: Password: dzk771w4k8 2024-09-17T09:17:11.244 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-09-17T09:17:11.244 INFO:teuthology.orchestra.run.smithi029.stderr:You can access the Ceph CLI with: 2024-09-17T09:17:11.244 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-09-17T09:17:11.244 INFO:teuthology.orchestra.run.smithi029.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:17:11.244 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-09-17T09:17:11.244 INFO:teuthology.orchestra.run.smithi029.stderr:Please consider enabling telemetry to help improve Ceph: 2024-09-17T09:17:11.245 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-09-17T09:17:11.245 INFO:teuthology.orchestra.run.smithi029.stderr: ceph telemetry on 2024-09-17T09:17:11.245 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-09-17T09:17:11.245 INFO:teuthology.orchestra.run.smithi029.stderr:For more information see: 2024-09-17T09:17:11.245 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-09-17T09:17:11.245 INFO:teuthology.orchestra.run.smithi029.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-09-17T09:17:11.245 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-09-17T09:17:11.245 INFO:teuthology.orchestra.run.smithi029.stderr:Bootstrap complete. 2024-09-17T09:17:11.245 INFO:teuthology.orchestra.run.smithi029.stderr:Releasing lock 139727085512736 on /run/cephadm/69135c5a-74d5-11ef-bceb-c7b262605968.lock 2024-09-17T09:17:11.245 INFO:teuthology.orchestra.run.smithi029.stderr:Lock 139727085512736 released on /run/cephadm/69135c5a-74d5-11ef-bceb-c7b262605968.lock 2024-09-17T09:17:11.278 INFO:tasks.cephadm:Fetching config... 2024-09-17T09:17:11.278 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-17T09:17:11.278 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-17T09:17:11.295 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-17T09:17:11.295 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-17T09:17:11.295 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-17T09:17:11.351 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-17T09:17:11.352 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-17T09:17:11.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-17T09:17:11.419 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-17T09:17:11.419 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-17T09:17:11.419 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-17T09:17:11.475 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-17T09:17:11.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8S6QmDaZ+moU3SA7wBy94ACGcgcxEJJZg6cooVj0JRa8FLUrbdWN/beGdj/Gc41m+svwHNyLH7RAIr72rwQtg4tUncLmGlp22+JeBOApdhb4B0owbyzKIYTJWbawkbVDPegbO0nmGt5srT8JTQfhHRceIN99HwIy+xxtv0/BOBhVeyho7NvaC/mQNve940U9PrMoY9b+nqG6ThTf6gyYHLzoOxtUqYflwtvcQfDVHeShAs6taIxxL9mL9+KuqYilBCKFJW32lp0vhrABqY1z8SAR0d8IlM6LSHazYWQc9ZIOmY8/TeOvrwh2Kc4c6AgIn8LkSudGlmQ9jw3J9p9r17HEthI7eNFCh3mUDe4XpsQtw0LXD4w39pPPnBUntqE/r4aTCfUk7bt+NlYX7AtK4Jd5nCaTTbkcKi98QKOOlsGNz0JvIqrYAmuE3U+w8ufPjoaKkpYbaxve+6bzI7RZNycLEzN8H7yb8RY9dRykc3A2rpJwtLmfcfIbcAbkc40c= ceph-69135c5a-74d5-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T09:17:11.518 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:10.640740+0000 mon.a (mon.0) 88 : audit [DBG] from='client.? 172.21.15.29:0/688130195' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-17T09:17:11.551 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8S6QmDaZ+moU3SA7wBy94ACGcgcxEJJZg6cooVj0JRa8FLUrbdWN/beGdj/Gc41m+svwHNyLH7RAIr72rwQtg4tUncLmGlp22+JeBOApdhb4B0owbyzKIYTJWbawkbVDPegbO0nmGt5srT8JTQfhHRceIN99HwIy+xxtv0/BOBhVeyho7NvaC/mQNve940U9PrMoY9b+nqG6ThTf6gyYHLzoOxtUqYflwtvcQfDVHeShAs6taIxxL9mL9+KuqYilBCKFJW32lp0vhrABqY1z8SAR0d8IlM6LSHazYWQc9ZIOmY8/TeOvrwh2Kc4c6AgIn8LkSudGlmQ9jw3J9p9r17HEthI7eNFCh3mUDe4XpsQtw0LXD4w39pPPnBUntqE/r4aTCfUk7bt+NlYX7AtK4Jd5nCaTTbkcKi98QKOOlsGNz0JvIqrYAmuE3U+w8ufPjoaKkpYbaxve+6bzI7RZNycLEzN8H7yb8RY9dRykc3A2rpJwtLmfcfIbcAbkc40c= ceph-69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:17:11.562 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8S6QmDaZ+moU3SA7wBy94ACGcgcxEJJZg6cooVj0JRa8FLUrbdWN/beGdj/Gc41m+svwHNyLH7RAIr72rwQtg4tUncLmGlp22+JeBOApdhb4B0owbyzKIYTJWbawkbVDPegbO0nmGt5srT8JTQfhHRceIN99HwIy+xxtv0/BOBhVeyho7NvaC/mQNve940U9PrMoY9b+nqG6ThTf6gyYHLzoOxtUqYflwtvcQfDVHeShAs6taIxxL9mL9+KuqYilBCKFJW32lp0vhrABqY1z8SAR0d8IlM6LSHazYWQc9ZIOmY8/TeOvrwh2Kc4c6AgIn8LkSudGlmQ9jw3J9p9r17HEthI7eNFCh3mUDe4XpsQtw0LXD4w39pPPnBUntqE/r4aTCfUk7bt+NlYX7AtK4Jd5nCaTTbkcKi98QKOOlsGNz0JvIqrYAmuE3U+w8ufPjoaKkpYbaxve+6bzI7RZNycLEzN8H7yb8RY9dRykc3A2rpJwtLmfcfIbcAbkc40c= ceph-69135c5a-74d5-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T09:17:11.635 INFO:teuthology.orchestra.run.smithi103.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8S6QmDaZ+moU3SA7wBy94ACGcgcxEJJZg6cooVj0JRa8FLUrbdWN/beGdj/Gc41m+svwHNyLH7RAIr72rwQtg4tUncLmGlp22+JeBOApdhb4B0owbyzKIYTJWbawkbVDPegbO0nmGt5srT8JTQfhHRceIN99HwIy+xxtv0/BOBhVeyho7NvaC/mQNve940U9PrMoY9b+nqG6ThTf6gyYHLzoOxtUqYflwtvcQfDVHeShAs6taIxxL9mL9+KuqYilBCKFJW32lp0vhrABqY1z8SAR0d8IlM6LSHazYWQc9ZIOmY8/TeOvrwh2Kc4c6AgIn8LkSudGlmQ9jw3J9p9r17HEthI7eNFCh3mUDe4XpsQtw0LXD4w39pPPnBUntqE/r4aTCfUk7bt+NlYX7AtK4Jd5nCaTTbkcKi98QKOOlsGNz0JvIqrYAmuE3U+w8ufPjoaKkpYbaxve+6bzI7RZNycLEzN8H7yb8RY9dRykc3A2rpJwtLmfcfIbcAbkc40c= ceph-69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:17:11.647 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi103 2024-09-17T09:17:11.648 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-17T09:17:11.648 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/etc/ceph/ceph.conf 2024-09-17T09:17:11.665 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-17T09:17:11.665 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:17:11.722 INFO:tasks.cephadm:Adding host smithi103 to orchestrator... 2024-09-17T09:17:11.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch host add smithi103 2024-09-17T09:17:12.704 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:11.434606+0000 mon.a (mon.0) 89 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-09-17T09:17:14.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:13.452394+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:14.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:13.804560+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:14.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:13.805607+0000 mon.a (mon.0) 92 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:14.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:13.808417+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:15.224 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:17:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: Warning: Permanently added 'smithi103,172.21.15.103' (ECDSA) to the list of known hosts. 2024-09-17T09:17:15.428 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi103' 2024-09-17T09:17:15.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:14.799195+0000 mgr.y (mgr.14144) 5 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi103", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:17:15.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:15.426739+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:15.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:15.427370+0000 mon.a (mon.0) 95 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:16.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-17T09:17:16.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:17:15.427106+0000 mgr.y (mgr.14144) 6 : cephadm [INF] Added host smithi103 2024-09-17T09:17:16.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:15.618634+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:16.969 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:15.823230+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:17.893 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:17:17.893 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "smithi029", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "smithi103", "hostname": "smithi103", "labels": [], "status": ""}] 2024-09-17T09:17:18.495 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-17T09:17:18.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-17T09:17:19.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:17.892117+0000 mgr.y (mgr.14144) 7 : audit [DBG] from='client.14162 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:17:19.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:18.803711+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:20.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:19.042483+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:20.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:19.044508+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:21.050 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-09-17T09:17:21.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:20.172720+0000 mon.a (mon.0) 101 : audit [INF] from='client.? 172.21.15.29:0/87733897' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-17T09:17:21.656 INFO:tasks.cephadm:Adding mon.a on smithi029 2024-09-17T09:17:21.656 INFO:tasks.cephadm:Adding mon.c on smithi029 2024-09-17T09:17:21.657 INFO:tasks.cephadm:Adding mon.b on smithi103 2024-09-17T09:17:21.657 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi029:172.21.15.29=a;smithi029:[v2:172.21.15.29:3301,v1:172.21.15.29:6790]=c;smithi103:172.21.15.103=b' 2024-09-17T09:17:22.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:21.045797+0000 mon.a (mon.0) 102 : audit [INF] from='client.? 172.21.15.29:0/87733897' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-17T09:17:22.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:21.045858+0000 mon.a (mon.0) 103 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:17:23.465 INFO:teuthology.orchestra.run.smithi103.stdout:Scheduled mon update... 2024-09-17T09:17:23.916 DEBUG:teuthology.orchestra.run.smithi029:mon.c> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.c.service 2024-09-17T09:17:23.919 DEBUG:teuthology.orchestra.run.smithi103:mon.b> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.b.service 2024-09-17T09:17:23.922 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T09:17:23.923 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T09:17:24.718 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:23.460365+0000 mgr.y (mgr.14144) 8 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi029:[v2:172.21.15.29:3301,v1:172.21.15.29:6790]=c;smithi103:172.21.15.103=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:17:24.718 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:17:23.461852+0000 mgr.y (mgr.14144) 9 : cephadm [INF] Saving service mon spec with placement smithi029:172.21.15.29=a;smithi029:[v2:172.21.15.29:3301,v1:172.21.15.29:6790]=c;smithi103:172.21.15.103=b;count:3 2024-09-17T09:17:24.718 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:23.464553+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:24.718 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:23.465268+0000 mon.a (mon.0) 105 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:24.718 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:23.469054+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:24.718 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:23.470151+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:17:24.718 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:23.470927+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:24.718 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:17:23.471862+0000 mgr.y (mgr.14144) 10 : cephadm [INF] Deploying daemon mon.c on smithi029 2024-09-17T09:17:25.735 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-17T09:17:25.735 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"fsid":"69135c5a-74d5-11ef-bceb-c7b262605968","modified":"2024-09-17T09:16:18.701305Z","created":"2024-09-17T09:16:18.701305Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-17T09:17:25.737 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 1 2024-09-17T09:17:25.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:24.537523+0000 mgr.y (mgr.14144) 11 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:26.729 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:25.734094+0000 mon.a (mon.0) 109 : audit [DBG] from='client.? 172.21.15.103:0/276808266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:17:26.730 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-17T09:17:26.730 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-17T09:17:26.730 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.allow_fallocate: 1 2024-09-17T09:17:26.730 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-17T09:17:26.730 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-17T09:17:26.730 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.use_direct_reads: 0 2024-09-17T09:17:26.730 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T09:17:26.731 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.create_missing_column_families: 0 2024-09-17T09:17:26.731 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.db_log_dir: 2024-09-17T09:17:26.731 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-09-17T09:17:26.731 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T09:17:26.731 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.max_subcompactions: 1 2024-09-17T09:17:26.731 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.max_background_flushes: -1 2024-09-17T09:17:26.731 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T09:17:26.731 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T09:17:26.731 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T09:17:26.731 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T09:17:26.731 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T09:17:26.731 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.advise_random_on_open: 1 2024-09-17T09:17:26.731 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-17T09:17:26.732 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.write_buffer_manager: 0x55b99d177ec0 2024-09-17T09:17:26.733 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T09:17:26.733 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-17T09:17:26.733 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T09:17:26.733 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T09:17:26.733 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.rate_limiter: (nil) 2024-09-17T09:17:26.733 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T09:17:26.733 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-17T09:17:26.733 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-17T09:17:26.733 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-17T09:17:26.733 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.unordered_write: 0 2024-09-17T09:17:26.733 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T09:17:26.733 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T09:17:26.734 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T09:17:26.734 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T09:17:26.734 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.row_cache: None 2024-09-17T09:17:26.734 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.wal_filter: None 2024-09-17T09:17:26.734 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T09:17:26.734 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-17T09:17:26.734 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.preserve_deletes: 0 2024-09-17T09:17:26.734 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.two_write_queues: 0 2024-09-17T09:17:26.734 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.manual_wal_flush: 0 2024-09-17T09:17:26.734 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.atomic_flush: 0 2024-09-17T09:17:26.734 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T09:17:26.734 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T09:17:26.734 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.log_readahead_size: 0 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.max_background_jobs: 2 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.max_background_compactions: -1 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.max_total_wal_size: 0 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.726+0000 7f593d241700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_open_files: -1 2024-09-17T09:17:26.735 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.bytes_per_sync: 0 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Compression algorithms supported: 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: kZSTD supported: 0 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: kXpressCompression supported: 0 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: kLZ4Compression supported: 1 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: kBZip2Compression supported: 0 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: kZlibCompression supported: 1 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: kSnappyCompression supported: 1 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T09:17:26.736 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 2024-09-17T09:17:26.737 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:26.737 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-09-17T09:17:26.737 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:26.737 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T09:17:26.737 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.merge_operator: 2024-09-17T09:17:26.737 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compaction_filter: None 2024-09-17T09:17:26.737 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compaction_filter_factory: None 2024-09-17T09:17:26.737 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T09:17:26.737 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T09:17:26.737 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b99d0860d8) 2024-09-17T09:17:26.737 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cache_index_and_filter_blocks: 1 2024-09-17T09:17:26.737 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T09:17:26.738 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T09:17:26.738 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: pin_top_level_index_and_filter: 1 2024-09-17T09:17:26.738 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: index_type: 0 2024-09-17T09:17:26.738 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: data_block_index_type: 0 2024-09-17T09:17:26.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: index_shortening: 1 2024-09-17T09:17:26.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: data_block_hash_table_util_ratio: 0.750000 2024-09-17T09:17:26.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: hash_index_allow_collision: 1 2024-09-17T09:17:26.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: checksum: 1 2024-09-17T09:17:26.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: no_block_cache: 0 2024-09-17T09:17:26.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: block_cache: 0x55b99d0bcf10 2024-09-17T09:17:26.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: block_cache_name: BinnedLRUCache 2024-09-17T09:17:26.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: block_cache_options: 2024-09-17T09:17:26.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: capacity : 536870912 2024-09-17T09:17:26.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: num_shard_bits : 4 2024-09-17T09:17:26.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: strict_capacity_limit : 0 2024-09-17T09:17:26.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: high_pri_pool_ratio: 0.000 2024-09-17T09:17:26.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: block_cache_compressed: (nil) 2024-09-17T09:17:27.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: persistent_cache: (nil) 2024-09-17T09:17:27.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: block_size: 4096 2024-09-17T09:17:27.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: block_size_deviation: 10 2024-09-17T09:17:27.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: block_restart_interval: 16 2024-09-17T09:17:27.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: index_block_restart_interval: 1 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: metadata_block_size: 4096 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: partition_filters: 0 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: use_delta_encoding: 1 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: whole_key_filtering: 1 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: verify_compression: 0 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: read_amp_bytes_per_bit: 0 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: format_version: 2 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: enable_index_compression: 1 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: block_align: 0 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-17T09:17:27.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compression: NoCompression 2024-09-17T09:17:27.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-17T09:17:27.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-17T09:17:27.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T09:17:27.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.num_levels: 7 2024-09-17T09:17:27.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T09:17:27.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T09:17:27.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T09:17:27.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T09:17:27.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T09:17:27.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T09:17:27.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T09:17:27.052 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:17:27.052 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T09:17:27.052 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T09:17:27.052 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compression_opts.level: 32767 2024-09-17T09:17:27.052 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-17T09:17:27.052 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T09:17:27.052 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:17:27.052 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compression_opts.enabled: false 2024-09-17T09:17:27.052 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T09:17:27.052 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T09:17:27.052 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T09:17:27.052 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T09:17:27.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.arena_block_size: 4194304 2024-09-17T09:17:27.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T09:17:27.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T09:17:27.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-17T09:17:27.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-17T09:17:27.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T09:17:27.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T09:17:27.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T09:17:27.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T09:17:27.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T09:17:27.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T09:17:27.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T09:17:27.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T09:17:27.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.table_properties_collectors: 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.inplace_update_support: 0 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.bloom_locality: 0 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.max_successive_merges: 0 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.force_consistency_checks: 0 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-17T09:17:27.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.ttl: 2592000 2024-09-17T09:17:27.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.727+0000 7f593d241700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T09:17:27.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.728+0000 7f593d241700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T09:17:27.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:27.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.728+0000 7f593d241700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-09-17T09:17:27.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:27.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.728+0000 7f593d241700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726564646728879, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-09-17T09:17:27.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.728+0000 7f593d241700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-09-17T09:17:27.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.728+0000 7f593d241700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.728+0000 7f593d241700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726564646729350, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726564646, "oldest_key_time": 3, "file_creation_time": 0}} 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.728+0000 7f593d241700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.729+0000 7f593d241700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726564646729864, "job": 1, "event": "recovery_finished"} 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.729+0000 7f593d241700 4 rocksdb: DB pointer 0x55b99d181800 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.730+0000 7f592405b700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.730+0000 7f592405b700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ** DB Stats ** 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T09:17:27.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ** Compaction Stats [default] ** 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ** Compaction Stats [default] ** 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:27.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:17:27.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T09:17:27.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:17:27.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:17:27.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:17:27.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:17:27.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Cumulative compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:17:27.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Interval compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:17:27.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:17:27.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:27.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:17:27.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:27.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ** Compaction Stats [default] ** 2024-09-17T09:17:27.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:17:27.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:17:27.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:27.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:27.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T09:17:27.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ** Compaction Stats [default] ** 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.3 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Cumulative compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:17:27.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:27.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:17:27.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:27.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.730+0000 7f593d241700 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-17T09:17:27.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.730+0000 7f593d241700 0 using public_addrv [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2024-09-17T09:17:27.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.731+0000 7f593d241700 0 starting mon.c rank -1 at public addrs [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] at bind addrs [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:17:27.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.732+0000 7f593d241700 1 mon.c@-1(???) e0 preinit fsid 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:17:27.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.746+0000 7f5926860700 0 mon.c@-1(synchronizing).mds e1 new map 2024-09-17T09:17:27.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.746+0000 7f5926860700 0 mon.c@-1(synchronizing).mds e1 print_map 2024-09-17T09:17:27.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: e1 2024-09-17T09:17:27.063 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T09:17:27.063 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T09:17:27.064 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: legacy client fscid: -1 2024-09-17T09:17:27.064 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:27.064 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: No filesystems configured 2024-09-17T09:17:27.064 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:17:27.064 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.746+0000 7f5926860700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-17T09:17:27.064 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.747+0000 7f5926860700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-17T09:17:27.064 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.747+0000 7f5926860700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-17T09:17:27.064 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.747+0000 7f5926860700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-17T09:17:27.064 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.747+0000 7f5926860700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-17T09:17:27.064 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.747+0000 7f5926860700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-17T09:17:27.064 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.747+0000 7f5926860700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T09:17:27.064 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.747+0000 7f5926860700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.747+0000 7f5926860700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.747+0000 7f5926860700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:25.734094+0000 mon.a (mon.0) 109 : audit [DBG] from='client.? 172.21.15.103:0/276808266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.747+0000 7f5926860700 1 mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.749+0000 7f5926860700 20 expand_channel_meta expand map: {default=false} 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.749+0000 7f5926860700 20 expand_channel_meta from 'false' to 'false' 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.749+0000 7f5926860700 20 expand_channel_meta expanded map: {default=false} 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.749+0000 7f5926860700 20 expand_channel_meta expand map: {default=info} 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.749+0000 7f5926860700 20 expand_channel_meta from 'info' to 'info' 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.749+0000 7f5926860700 20 expand_channel_meta expanded map: {default=info} 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.749+0000 7f5926860700 20 expand_channel_meta expand map: {default=daemon} 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.749+0000 7f5926860700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.749+0000 7f5926860700 20 expand_channel_meta expanded map: {default=daemon} 2024-09-17T09:17:27.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.749+0000 7f5926860700 20 expand_channel_meta expand map: {default=debug} 2024-09-17T09:17:27.066 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.749+0000 7f5926860700 20 expand_channel_meta from 'debug' to 'debug' 2024-09-17T09:17:27.066 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.749+0000 7f5926860700 20 expand_channel_meta expanded map: {default=debug} 2024-09-17T09:17:27.066 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:17:26.749+0000 7f5926860700 10 mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2024-09-17T09:17:27.066 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 bash[26923]: aa05f3e5656a941e7d8d4d0a8835726559f8a8f54bdd8b7bdc2d55180184605d 2024-09-17T09:17:27.066 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:26 smithi029 systemd[1]: Started Ceph mon.c for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:17:27.303 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T09:17:27.303 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T09:17:32.064 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-17T09:17:32.065 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":2,"fsid":"69135c5a-74d5-11ef-bceb-c7b262605968","modified":"2024-09-17T09:17:26.752521Z","created":"2024-09-17T09:16:18.701305Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3301","nonce":0},{"type":"v1","addr":"172.21.15.29:6790","nonce":0}]},"addr":"172.21.15.29:6790/0","public_addr":"172.21.15.29:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T09:17:32.066 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 2 2024-09-17T09:17:32.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:26.537815+0000 mgr.y (mgr.14144) 12 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:32.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:26.754540+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:17:32.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:26.754732+0000 mon.a (mon.0) 112 : cluster [INF] mon.a calling monitor election 2024-09-17T09:17:32.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:26.755706+0000 mon.a (mon.0) 113 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:27.752123+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:28.538093+0000 mgr.y (mgr.14144) 13 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:28.752259+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:28.753462+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2024-09-17T09:17:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:29.752335+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:30.538306+0000 mgr.y (mgr.14144) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:30.752457+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:31.752537+0000 mon.a (mon.0) 118 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:31.757186+0000 mon.a (mon.0) 119 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T09:17:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:31.759945+0000 mon.a (mon.0) 120 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} 2024-09-17T09:17:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:31.759999+0000 mon.a (mon.0) 121 : cluster [DBG] fsmap 2024-09-17T09:17:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:31.760032+0000 mon.a (mon.0) 122 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:17:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:31.760222+0000 mon.a (mon.0) 123 : cluster [DBG] mgrmap e13: y(active, since 27s) 2024-09-17T09:17:32.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:31.763394+0000 mon.a (mon.0) 124 : cluster [INF] overall HEALTH_OK 2024-09-17T09:17:32.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:26.537815+0000 mgr.y (mgr.14144) 12 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:32.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:26.754540+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:17:32.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:26.754732+0000 mon.a (mon.0) 112 : cluster [INF] mon.a calling monitor election 2024-09-17T09:17:32.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:26.755706+0000 mon.a (mon.0) 113 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:32.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:27.752123+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:32.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:28.538093+0000 mgr.y (mgr.14144) 13 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:32.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:28.752259+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:32.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:28.753462+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2024-09-17T09:17:32.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:29.752335+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:32.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:30.538306+0000 mgr.y (mgr.14144) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:32.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:30.752457+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:32.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:31.752537+0000 mon.a (mon.0) 118 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:32.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:31.757186+0000 mon.a (mon.0) 119 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T09:17:32.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:31.759945+0000 mon.a (mon.0) 120 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} 2024-09-17T09:17:32.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:31.759999+0000 mon.a (mon.0) 121 : cluster [DBG] fsmap 2024-09-17T09:17:32.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:31.760032+0000 mon.a (mon.0) 122 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:17:32.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:31.760222+0000 mon.a (mon.0) 123 : cluster [DBG] mgrmap e13: y(active, since 27s) 2024-09-17T09:17:32.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:31.763394+0000 mon.a (mon.0) 124 : cluster [INF] overall HEALTH_OK 2024-09-17T09:17:32.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:31.765434+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:32.302 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:31.766327+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:17:32.302 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:31.767225+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:32.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:31.765434+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:32.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:31.766327+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:17:32.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:31.767225+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:33.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:17:31.767919+0000 mgr.y (mgr.14144) 15 : cephadm [INF] Deploying daemon mon.b on smithi103 2024-09-17T09:17:33.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:32.063559+0000 mon.a (mon.0) 128 : audit [DBG] from='client.? 172.21.15.103:0/1702060563' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:17:33.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:32.752947+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:33.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:17:31.767919+0000 mgr.y (mgr.14144) 15 : cephadm [INF] Deploying daemon mon.b on smithi103 2024-09-17T09:17:33.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:32.063559+0000 mon.a (mon.0) 128 : audit [DBG] from='client.? 172.21.15.103:0/1702060563' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:17:33.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:32.752947+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:33.564 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T09:17:33.565 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T09:17:34.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:32.538662+0000 mgr.y (mgr.14144) 16 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:34.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:32.538662+0000 mgr.y (mgr.14144) 16 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:36.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:34.538928+0000 mgr.y (mgr.14144) 17 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:34.538928+0000 mgr.y (mgr.14144) 17 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:36.639 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-17T09:17:36.639 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":2,"fsid":"69135c5a-74d5-11ef-bceb-c7b262605968","modified":"2024-09-17T09:17:26.752521Z","created":"2024-09-17T09:16:18.701305Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3301","nonce":0},{"type":"v1","addr":"172.21.15.29:6790","nonce":0}]},"addr":"172.21.15.29:6790/0","public_addr":"172.21.15.29:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T09:17:36.641 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 2 2024-09-17T09:17:37.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:36.638006+0000 mon.a (mon.0) 130 : audit [DBG] from='client.? 172.21.15.103:0/664435459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:17:37.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:36.638006+0000 mon.a (mon.0) 130 : audit [DBG] from='client.? 172.21.15.103:0/664435459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:17:37.574 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 podman[23870]: 2024-09-17 09:17:37.375719276 +0000 UTC m=+0.015556548 image pull 24ecd6d5f14c2268c5d2c666aa151456949cf76c6747d873c6fcf41f6f672d15 quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000 2024-09-17T09:17:38.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 podman[23870]: 2024-09-17 09:17:37.660361497 +0000 UTC m=+0.300198760 container init 203d1f26d7b87eb313b794f6ec7b722b3a1300f8097a11f1025a2662ca17f10c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b, org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, RELEASE=HEAD, CEPH_POINT_RELEASE=-16.2.0, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20201204) 2024-09-17T09:17:38.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 podman[23870]: 2024-09-17 09:17:37.663651757 +0000 UTC m=+0.303489026 container start 203d1f26d7b87eb313b794f6ec7b722b3a1300f8097a11f1025a2662ca17f10c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-16.2.0, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, ceph=True, RELEASE=HEAD) 2024-09-17T09:17:38.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.682+0000 7fd2c5704700 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-17T09:17:38.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.682+0000 7fd2c5704700 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-mon, pid 2 2024-09-17T09:17:38.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.682+0000 7fd2c5704700 0 pidfile_write: ignore empty --pid-file 2024-09-17T09:17:38.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.687+0000 7fd2c5704700 0 load: jerasure load: lrc load: isa 2024-09-17T09:17:38.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: RocksDB version: 6.8.1 2024-09-17T09:17:38.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-17T09:17:38.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Compile date Mar 30 2021 2024-09-17T09:17:38.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: DB SUMMARY 2024-09-17T09:17:38.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: CURRENT file: CURRENT 2024-09-17T09:17:38.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: IDENTITY file: IDENTITY 2024-09-17T09:17:38.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-09-17T09:17:38.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-17T09:17:38.010 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.010 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000003.log size: 636 ; 2024-09-17T09:17:38.010 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.010 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.error_if_exists: 0 2024-09-17T09:17:38.010 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.create_if_missing: 0 2024-09-17T09:17:38.011 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.paranoid_checks: 1 2024-09-17T09:17:38.011 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.env: 0x55ee9d79d1c0 2024-09-17T09:17:38.011 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.fs: Posix File System 2024-09-17T09:17:38.011 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.info_log: 0x55ee9fd16640 2024-09-17T09:17:38.011 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-17T09:17:38.011 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.statistics: (nil) 2024-09-17T09:17:38.011 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.use_fsync: 0 2024-09-17T09:17:38.011 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_log_file_size: 0 2024-09-17T09:17:38.011 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T09:17:38.012 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T09:17:38.012 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-17T09:17:38.012 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-17T09:17:38.012 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.allow_fallocate: 1 2024-09-17T09:17:38.012 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-17T09:17:38.012 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-17T09:17:38.012 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.use_direct_reads: 0 2024-09-17T09:17:38.012 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T09:17:38.012 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.create_missing_column_families: 0 2024-09-17T09:17:38.012 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.db_log_dir: 2024-09-17T09:17:38.012 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-09-17T09:17:38.013 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T09:17:38.013 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_subcompactions: 1 2024-09-17T09:17:38.013 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_background_flushes: -1 2024-09-17T09:17:38.013 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T09:17:38.013 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T09:17:38.014 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T09:17:38.014 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T09:17:38.014 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T09:17:38.014 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.advise_random_on_open: 1 2024-09-17T09:17:38.014 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-17T09:17:38.014 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.write_buffer_manager: 0x55ee9fdade30 2024-09-17T09:17:38.014 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T09:17:38.014 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-17T09:17:38.014 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T09:17:38.014 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T09:17:38.014 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.rate_limiter: (nil) 2024-09-17T09:17:38.015 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T09:17:38.015 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-17T09:17:38.015 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-17T09:17:38.015 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-17T09:17:38.015 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.unordered_write: 0 2024-09-17T09:17:38.015 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T09:17:38.015 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T09:17:38.015 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T09:17:38.015 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T09:17:38.015 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.row_cache: None 2024-09-17T09:17:38.015 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.wal_filter: None 2024-09-17T09:17:38.016 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T09:17:38.016 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-17T09:17:38.016 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.preserve_deletes: 0 2024-09-17T09:17:38.016 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.two_write_queues: 0 2024-09-17T09:17:38.016 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.manual_wal_flush: 0 2024-09-17T09:17:38.016 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.atomic_flush: 0 2024-09-17T09:17:38.016 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T09:17:38.016 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T09:17:38.016 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T09:17:38.016 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.log_readahead_size: 0 2024-09-17T09:17:38.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-09-17T09:17:38.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_background_jobs: 2 2024-09-17T09:17:38.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_background_compactions: -1 2024-09-17T09:17:38.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T09:17:38.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T09:17:38.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T09:17:38.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_total_wal_size: 0 2024-09-17T09:17:38.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T09:17:38.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T09:17:38.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T09:17:38.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T09:17:38.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_open_files: -1 2024-09-17T09:17:38.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.bytes_per_sync: 0 2024-09-17T09:17:38.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T09:17:38.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T09:17:38.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-17T09:17:38.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Compression algorithms supported: 2024-09-17T09:17:38.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T09:17:38.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: kZSTD supported: 0 2024-09-17T09:17:38.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: kXpressCompression supported: 0 2024-09-17T09:17:38.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-17T09:17:38.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: kLZ4Compression supported: 1 2024-09-17T09:17:38.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: kBZip2Compression supported: 0 2024-09-17T09:17:38.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: kZlibCompression supported: 1 2024-09-17T09:17:38.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: kSnappyCompression supported: 1 2024-09-17T09:17:38.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T09:17:38.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 2024-09-17T09:17:38.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-09-17T09:17:38.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T09:17:38.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.merge_operator: 2024-09-17T09:17:38.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compaction_filter: None 2024-09-17T09:17:38.020 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compaction_filter_factory: None 2024-09-17T09:17:38.020 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T09:17:38.020 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T09:17:38.020 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ee9fcbc0d8) 2024-09-17T09:17:38.020 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cache_index_and_filter_blocks: 1 2024-09-17T09:17:38.020 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T09:17:38.020 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T09:17:38.020 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: pin_top_level_index_and_filter: 1 2024-09-17T09:17:38.020 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: index_type: 0 2024-09-17T09:17:38.020 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: data_block_index_type: 0 2024-09-17T09:17:38.020 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: index_shortening: 1 2024-09-17T09:17:38.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: data_block_hash_table_util_ratio: 0.750000 2024-09-17T09:17:38.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: hash_index_allow_collision: 1 2024-09-17T09:17:38.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: checksum: 1 2024-09-17T09:17:38.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: no_block_cache: 0 2024-09-17T09:17:38.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: block_cache: 0x55ee9fcf2f10 2024-09-17T09:17:38.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: block_cache_name: BinnedLRUCache 2024-09-17T09:17:38.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: block_cache_options: 2024-09-17T09:17:38.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: capacity : 536870912 2024-09-17T09:17:38.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: num_shard_bits : 4 2024-09-17T09:17:38.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: strict_capacity_limit : 0 2024-09-17T09:17:38.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: high_pri_pool_ratio: 0.000 2024-09-17T09:17:38.022 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: block_cache_compressed: (nil) 2024-09-17T09:17:38.022 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: persistent_cache: (nil) 2024-09-17T09:17:38.022 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: block_size: 4096 2024-09-17T09:17:38.022 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: block_size_deviation: 10 2024-09-17T09:17:38.022 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: block_restart_interval: 16 2024-09-17T09:17:38.022 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: index_block_restart_interval: 1 2024-09-17T09:17:38.022 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: metadata_block_size: 4096 2024-09-17T09:17:38.022 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: partition_filters: 0 2024-09-17T09:17:38.022 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: use_delta_encoding: 1 2024-09-17T09:17:38.022 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-17T09:17:38.023 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: whole_key_filtering: 1 2024-09-17T09:17:38.023 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: verify_compression: 0 2024-09-17T09:17:38.023 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: read_amp_bytes_per_bit: 0 2024-09-17T09:17:38.023 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: format_version: 2 2024-09-17T09:17:38.023 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: enable_index_compression: 1 2024-09-17T09:17:38.023 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: block_align: 0 2024-09-17T09:17:38.023 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.023 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-17T09:17:38.023 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-17T09:17:38.023 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compression: NoCompression 2024-09-17T09:17:38.023 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-17T09:17:38.023 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-17T09:17:38.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T09:17:38.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.num_levels: 7 2024-09-17T09:17:38.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T09:17:38.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T09:17:38.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T09:17:38.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T09:17:38.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T09:17:38.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T09:17:38.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T09:17:38.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:17:38.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T09:17:38.024 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T09:17:38.025 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compression_opts.level: 32767 2024-09-17T09:17:38.025 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-17T09:17:38.025 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T09:17:38.025 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:17:38.025 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compression_opts.enabled: false 2024-09-17T09:17:38.025 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T09:17:38.025 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T09:17:38.025 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T09:17:38.025 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-17T09:17:38.025 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T09:17:38.025 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T09:17:38.025 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T09:17:38.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T09:17:38.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T09:17:38.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T09:17:38.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T09:17:38.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T09:17:38.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T09:17:38.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T09:17:38.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T09:17:38.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T09:17:38.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T09:17:38.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.arena_block_size: 4194304 2024-09-17T09:17:38.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T09:17:38.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T09:17:38.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-17T09:17:38.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-17T09:17:38.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T09:17:38.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T09:17:38.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T09:17:38.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T09:17:38.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T09:17:38.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T09:17:38.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T09:17:38.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T09:17:38.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T09:17:38.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T09:17:38.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.table_properties_collectors: 2024-09-17T09:17:38.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.inplace_update_support: 0 2024-09-17T09:17:38.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T09:17:38.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T09:17:38.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T09:17:38.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T09:17:38.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.bloom_locality: 0 2024-09-17T09:17:38.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.max_successive_merges: 0 2024-09-17T09:17:38.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T09:17:38.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-17T09:17:38.028 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.force_consistency_checks: 0 2024-09-17T09:17:38.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-17T09:17:38.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.ttl: 2592000 2024-09-17T09:17:38.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.688+0000 7fd2c5704700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T09:17:38.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.689+0000 7fd2c5704700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T09:17:38.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.689+0000 7fd2c5704700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-09-17T09:17:38.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.689+0000 7fd2c5704700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726564657690640, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-09-17T09:17:38.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.689+0000 7fd2c5704700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-09-17T09:17:38.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.689+0000 7fd2c5704700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-17T09:17:38.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.689+0000 7fd2c5704700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726564657691096, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1562, "table_properties": {"data_size": 648, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726564657, "oldest_key_time": 3, "file_creation_time": 0}} 2024-09-17T09:17:38.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.689+0000 7fd2c5704700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-09-17T09:17:38.030 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.030 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.690+0000 7fd2c5704700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1726564657691458, "job": 1, "event": "recovery_finished"} 2024-09-17T09:17:38.030 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.690+0000 7fd2c5704700 4 rocksdb: DB pointer 0x55ee9fdb7800 2024-09-17T09:17:38.030 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.690+0000 7fd2ac51e700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-09-17T09:17:38.030 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.690+0000 7fd2ac51e700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-09-17T09:17:38.030 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ** DB Stats ** 2024-09-17T09:17:38.030 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:17:38.030 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T09:17:38.030 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:17:38.030 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:17:38.030 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T09:17:38.031 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-17T09:17:38.032 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:17:38.032 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.032 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ** Compaction Stats [default] ** 2024-09-17T09:17:38.032 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:17:38.032 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:17:38.033 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:38.033 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:38.033 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:38.033 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.033 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ** Compaction Stats [default] ** 2024-09-17T09:17:38.033 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:17:38.033 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:17:38.033 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:38.033 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:17:38.033 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T09:17:38.033 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:17:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:17:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:17:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:17:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:17:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:17:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:17:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:17:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ** Compaction Stats [default] ** 2024-09-17T09:17:38.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:17:38.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:17:38.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:38.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:38.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T09:17:38.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ** Compaction Stats [default] ** 2024-09-17T09:17:38.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T09:17:38.035 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:17:38.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2024-09-17T09:17:38.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:17:38.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T09:17:38.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:17:38.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:17:38.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:17:38.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:17:38.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:17:38.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:17:38.036 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:17:38.037 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.037 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:17:38.037 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.037 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.690+0000 7fd2c5704700 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-17T09:17:38.037 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.691+0000 7fd2c5704700 0 using public_addr v2:172.21.15.103:0/0 -> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] 2024-09-17T09:17:38.037 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.691+0000 7fd2c5704700 0 starting mon.b rank -1 at public addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] at bind addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:17:38.037 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.692+0000 7fd2c5704700 1 mon.b@-1(???) e0 preinit fsid 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:17:38.037 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.707+0000 7fd2aed23700 0 mon.b@-1(synchronizing).mds e1 new map 2024-09-17T09:17:38.038 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.707+0000 7fd2aed23700 0 mon.b@-1(synchronizing).mds e1 print_map 2024-09-17T09:17:38.038 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: e1 2024-09-17T09:17:38.038 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T09:17:38.038 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T09:17:38.038 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: legacy client fscid: -1 2024-09-17T09:17:38.038 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.038 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: No filesystems configured 2024-09-17T09:17:38.038 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:17:38.039 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.707+0000 7fd2aed23700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-17T09:17:38.039 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.708+0000 7fd2aed23700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-17T09:17:38.039 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.708+0000 7fd2aed23700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-17T09:17:38.039 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.708+0000 7fd2aed23700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-17T09:17:38.039 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.708+0000 7fd2aed23700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-17T09:17:38.039 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.708+0000 7fd2aed23700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-17T09:17:38.039 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.708+0000 7fd2aed23700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T09:17:38.039 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.708+0000 7fd2aed23700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:17:38.039 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.708+0000 7fd2aed23700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:17:38.039 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.708+0000 7fd2aed23700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T09:17:38.039 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:36.638006+0000 mon.a (mon.0) 130 : audit [DBG] from='client.? 172.21.15.103:0/664435459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:17:38.039 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:17:37.708+0000 7fd2aed23700 1 mon.b@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-09-17T09:17:38.040 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 bash[23870]: 203d1f26d7b87eb313b794f6ec7b722b3a1300f8097a11f1025a2662ca17f10c 2024-09-17T09:17:38.040 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:37 smithi103 systemd[1]: Started Ceph mon.b for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:17:38.314 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T09:17:38.314 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T09:17:43.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:37.716460+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:17:43.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:37.716690+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:43.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:37.716841+0000 mon.a (mon.0) 134 : cluster [INF] mon.a calling monitor election 2024-09-17T09:17:43.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:37.717047+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2024-09-17T09:17:43.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:37.717956+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:43.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:38.539481+0000 mgr.y (mgr.14144) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:43.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:38.713042+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:43.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:39.713373+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:43.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:40.539818+0000 mgr.y (mgr.14144) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:43.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:40.713368+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:43.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:41.713527+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:43.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:42.713618+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:43.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:42.718067+0000 mon.a (mon.0) 141 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T09:17:43.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:42.720713+0000 mon.a (mon.0) 142 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} 2024-09-17T09:17:43.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:42.720758+0000 mon.a (mon.0) 143 : cluster [DBG] fsmap 2024-09-17T09:17:43.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:42.720788+0000 mon.a (mon.0) 144 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:17:43.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:42.720983+0000 mon.a (mon.0) 145 : cluster [DBG] mgrmap e13: y(active, since 38s) 2024-09-17T09:17:43.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:42.721162+0000 mon.a (mon.0) 146 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-17T09:17:43.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:42.724608+0000 mon.a (mon.0) 147 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-09-17T09:17:43.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:42.724640+0000 mon.a (mon.0) 148 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-17T09:17:43.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:42.724667+0000 mon.a (mon.0) 149 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] is down (out of quorum) 2024-09-17T09:17:43.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:42.726985+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:43.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:42.727739+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:43.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:37.716460+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:17:43.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:37.716690+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:43.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:37.716841+0000 mon.a (mon.0) 134 : cluster [INF] mon.a calling monitor election 2024-09-17T09:17:43.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:37.717047+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2024-09-17T09:17:43.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:37.717956+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:17:43.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:38.539481+0000 mgr.y (mgr.14144) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:43.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:38.713042+0000 mon.a (mon.0) 136 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:43.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:39.713373+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:43.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:40.539818+0000 mgr.y (mgr.14144) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:43.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:40.713368+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:43.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:41.713527+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:43.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:42.713618+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:43.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:42.718067+0000 mon.a (mon.0) 141 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T09:17:43.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:42.720713+0000 mon.a (mon.0) 142 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} 2024-09-17T09:17:43.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:42.720758+0000 mon.a (mon.0) 143 : cluster [DBG] fsmap 2024-09-17T09:17:43.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:42.720788+0000 mon.a (mon.0) 144 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:17:43.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:42.720983+0000 mon.a (mon.0) 145 : cluster [DBG] mgrmap e13: y(active, since 38s) 2024-09-17T09:17:43.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:42.721162+0000 mon.a (mon.0) 146 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-17T09:17:43.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:42.724608+0000 mon.a (mon.0) 147 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-09-17T09:17:43.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:42.724640+0000 mon.a (mon.0) 148 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-17T09:17:43.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:42.724667+0000 mon.a (mon.0) 149 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] is down (out of quorum) 2024-09-17T09:17:43.056 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:42.726985+0000 mon.a (mon.0) 150 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:43.057 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:42.727739+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:43.137 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-09-17T09:17:43.137 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":3,"fsid":"69135c5a-74d5-11ef-bceb-c7b262605968","modified":"2024-09-17T09:17:37.713247Z","created":"2024-09-17T09:16:18.701305Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3301","nonce":0},{"type":"v1","addr":"172.21.15.29:6790","nonce":0}]},"addr":"172.21.15.29:6790/0","public_addr":"172.21.15.29:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T09:17:43.139 INFO:teuthology.orchestra.run.smithi103.stderr:dumped monmap epoch 3 2024-09-17T09:17:43.781 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-17T09:17:43.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-17T09:17:44.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:42.540098+0000 mgr.y (mgr.14144) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:44.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:43.135978+0000 mon.a (mon.0) 152 : audit [DBG] from='client.? 172.21.15.103:0/725540140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:17:44.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:43.714072+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:44.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:43.756579+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:44.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:42.540098+0000 mgr.y (mgr.14144) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:44.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:43.135978+0000 mon.a (mon.0) 152 : audit [DBG] from='client.? 172.21.15.103:0/725540140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:17:44.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:43.714072+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:44.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:43.756579+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:45.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:42.540098+0000 mgr.y (mgr.14144) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:45.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:43.135978+0000 mon.a (mon.0) 152 : audit [DBG] from='client.? 172.21.15.103:0/725540140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T09:17:45.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:43.714072+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:45.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:43.756579+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:45.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:39.714273+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-09-17T09:17:45.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:44.716446+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2024-09-17T09:17:45.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:44.716999+0000 mon.c (mon.1) 3 : cluster [INF] mon.c calling monitor election 2024-09-17T09:17:45.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:44.717130+0000 mon.a (mon.0) 161 : cluster [INF] mon.a calling monitor election 2024-09-17T09:17:45.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:44.718786+0000 mon.a (mon.0) 162 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T09:17:45.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:44.721622+0000 mon.a (mon.0) 163 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} 2024-09-17T09:17:45.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:44.721668+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-09-17T09:17:45.406 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:44.721690+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:17:45.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:44.721850+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 40s) 2024-09-17T09:17:45.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:44.721997+0000 mon.a (mon.0) 167 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-17T09:17:45.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:44.722014+0000 mon.a (mon.0) 168 : cluster [INF] Cluster is now healthy 2024-09-17T09:17:45.407 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:44.724999+0000 mon.a (mon.0) 169 : cluster [INF] overall HEALTH_OK 2024-09-17T09:17:45.407 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:39.714273+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-09-17T09:17:45.407 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:44.716446+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2024-09-17T09:17:45.407 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:44.716999+0000 mon.c (mon.1) 3 : cluster [INF] mon.c calling monitor election 2024-09-17T09:17:45.407 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:44.717130+0000 mon.a (mon.0) 161 : cluster [INF] mon.a calling monitor election 2024-09-17T09:17:45.407 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:44.718786+0000 mon.a (mon.0) 162 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T09:17:45.407 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:44.721622+0000 mon.a (mon.0) 163 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} 2024-09-17T09:17:45.407 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:44.721668+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-09-17T09:17:45.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:44.721690+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:17:45.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:44.721850+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 40s) 2024-09-17T09:17:45.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:44.721997+0000 mon.a (mon.0) 167 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-17T09:17:45.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:44.722014+0000 mon.a (mon.0) 168 : cluster [INF] Cluster is now healthy 2024-09-17T09:17:45.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:44.724999+0000 mon.a (mon.0) 169 : cluster [INF] overall HEALTH_OK 2024-09-17T09:17:45.438 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:39.714273+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2024-09-17T09:17:45.438 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:44.716446+0000 mon.b (mon.2) 2 : cluster [INF] mon.b calling monitor election 2024-09-17T09:17:45.438 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:44.716999+0000 mon.c (mon.1) 3 : cluster [INF] mon.c calling monitor election 2024-09-17T09:17:45.438 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:44.717130+0000 mon.a (mon.0) 161 : cluster [INF] mon.a calling monitor election 2024-09-17T09:17:45.438 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:44.718786+0000 mon.a (mon.0) 162 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T09:17:45.438 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:44.721622+0000 mon.a (mon.0) 163 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} 2024-09-17T09:17:45.438 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:44.721668+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-09-17T09:17:45.439 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:44.721690+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-17T09:17:45.439 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:44.721850+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 40s) 2024-09-17T09:17:45.439 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:44.721997+0000 mon.a (mon.0) 167 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-17T09:17:45.439 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:44.722014+0000 mon.a (mon.0) 168 : cluster [INF] Cluster is now healthy 2024-09-17T09:17:45.439 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:44.724999+0000 mon.a (mon.0) 169 : cluster [INF] overall HEALTH_OK 2024-09-17T09:17:46.400 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:44.540394+0000 mgr.y (mgr.14144) 24 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:46.400 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:45.714118+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:46.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:44.540394+0000 mgr.y (mgr.14144) 24 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:46.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:45.714118+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:46.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:44.540394+0000 mgr.y (mgr.14144) 24 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:46.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:45.714118+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:17:47.238 INFO:teuthology.orchestra.run.smithi029.stdout:# minimal ceph.conf for 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:17:47.238 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-09-17T09:17:47.239 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:17:47.239 INFO:teuthology.orchestra.run.smithi029.stdout: mon_host = [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2024-09-17T09:17:47.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:46.143900+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:47.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:17:46.144449+0000 mgr.y (mgr.14144) 25 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-17T09:17:47.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:46.144722+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:17:47.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:46.145410+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:17:47.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:46.146073+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:47.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:17:46.146665+0000 mgr.y (mgr.14144) 26 : cephadm [INF] Reconfiguring daemon mon.c on smithi029 2024-09-17T09:17:47.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:46.981416+0000 mon.a (mon.0) 175 : audit [DBG] from='client.? 172.21.15.29:0/4150641607' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:47.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:46.143900+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:47.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:17:46.144449+0000 mgr.y (mgr.14144) 25 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-17T09:17:47.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:46.144722+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:17:47.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:46.145410+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:17:47.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:46.146073+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:47.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:17:46.146665+0000 mgr.y (mgr.14144) 26 : cephadm [INF] Reconfiguring daemon mon.c on smithi029 2024-09-17T09:17:47.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:46.981416+0000 mon.a (mon.0) 175 : audit [DBG] from='client.? 172.21.15.29:0/4150641607' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:47.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:46.143900+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:47.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:17:46.144449+0000 mgr.y (mgr.14144) 25 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-17T09:17:47.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:46.144722+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:17:47.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:46.145410+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:17:47.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:46.146073+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:47.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:17:46.146665+0000 mgr.y (mgr.14144) 26 : cephadm [INF] Reconfiguring daemon mon.c on smithi029 2024-09-17T09:17:47.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:46.981416+0000 mon.a (mon.0) 175 : audit [DBG] from='client.? 172.21.15.29:0/4150641607' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:47.801 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-17T09:17:47.802 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-17T09:17:47.802 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T09:17:47.831 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-17T09:17:47.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:17:47.899 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-17T09:17:47.900 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T09:17:47.934 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-17T09:17:47.934 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T09:17:48.003 INFO:tasks.cephadm:Adding mgr.y on smithi029 2024-09-17T09:17:48.003 INFO:tasks.cephadm:Adding mgr.x on smithi103 2024-09-17T09:17:48.003 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi029=y;smithi103=x' 2024-09-17T09:17:48.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:46.540795+0000 mgr.y (mgr.14144) 27 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:48.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:46.540795+0000 mgr.y (mgr.14144) 27 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:48.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:46.540795+0000 mgr.y (mgr.14144) 27 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:49.621 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:48.308910+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:49.790 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:17:48.309364+0000 mgr.y (mgr.14144) 28 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-17T09:17:49.791 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:48.309587+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:17:49.791 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:48.310218+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:17:49.791 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:48.310839+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:49.791 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:17:48.311462+0000 mgr.y (mgr.14144) 29 : cephadm [INF] Reconfiguring daemon mon.b on smithi103 2024-09-17T09:17:49.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:48.308910+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:49.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:17:48.309364+0000 mgr.y (mgr.14144) 28 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-17T09:17:49.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:48.309587+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:17:49.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:48.310218+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:17:49.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:48.310839+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:49.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:17:48.311462+0000 mgr.y (mgr.14144) 29 : cephadm [INF] Reconfiguring daemon mon.b on smithi103 2024-09-17T09:17:49.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:48.308910+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:49.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:17:48.309364+0000 mgr.y (mgr.14144) 28 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-17T09:17:49.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:48.309587+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:17:49.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:48.310218+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:17:49.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:48.310839+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:49.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:17:48.311462+0000 mgr.y (mgr.14144) 29 : cephadm [INF] Reconfiguring daemon mon.b on smithi103 2024-09-17T09:17:50.568 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:48.541155+0000 mgr.y (mgr.14144) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:50.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:48.541155+0000 mgr.y (mgr.14144) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:50.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:48.541155+0000 mgr.y (mgr.14144) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:50.971 INFO:teuthology.orchestra.run.smithi103.stdout:Scheduled mgr update... 2024-09-17T09:17:51.544 DEBUG:teuthology.orchestra.run.smithi103:mgr.x> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.x.service 2024-09-17T09:17:51.546 INFO:tasks.cephadm:Deploying OSDs... 2024-09-17T09:17:51.547 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-17T09:17:51.547 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T09:17:51.567 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T09:17:51.567 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-09-17T09:17:51.628 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T09:17:51.629 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:17:51.629 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-17T09:17:51.629 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:17:51.629 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:17:51.629 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-09-17 09:17:26.246381605 +0000 2024-09-17T09:17:51.629 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-09-17 09:15:20.998339469 +0000 2024-09-17T09:17:51.629 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-09-17 09:15:20.998339469 +0000 2024-09-17T09:17:51.629 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-09-17 09:15:20.998339469 +0000 2024-09-17T09:17:51.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T09:17:51.698 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-09-17T09:17:51.698 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-09-17T09:17:51.698 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000134746 s, 3.8 MB/s 2024-09-17T09:17:51.700 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T09:17:51.760 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-09-17T09:17:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T09:17:51.817 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:17:51.818 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-17T09:17:51.818 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:17:51.818 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:17:51.818 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-09-17 09:17:26.246381605 +0000 2024-09-17T09:17:51.818 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-09-17 09:15:21.324334104 +0000 2024-09-17T09:17:51.818 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-09-17 09:15:21.324334104 +0000 2024-09-17T09:17:51.818 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-09-17 09:15:21.324334104 +0000 2024-09-17T09:17:51.818 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T09:17:51.825 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:50.541529+0000 mgr.y (mgr.14144) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:51.825 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:50.622216+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:51.825 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:50.623018+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:51.825 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:50.628167+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:51.825 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:50.966191+0000 mgr.y (mgr.14144) 32 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi029=y;smithi103=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:17:51.825 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:17:50.966876+0000 mgr.y (mgr.14144) 33 : cephadm [INF] Saving service mgr spec with placement smithi029=y;smithi103=x;count:2 2024-09-17T09:17:51.825 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:50.970682+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:51.825 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:50.971350+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:51.825 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:50.976248+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:51.826 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:50.977241+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:17:51.826 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:50.979806+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T09:17:51.826 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:50.980558+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:17:51.826 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:50.981493+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:51.826 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:17:50.982177+0000 mgr.y (mgr.14144) 34 : cephadm [INF] Deploying daemon mgr.x on smithi103 2024-09-17T09:17:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:50.541529+0000 mgr.y (mgr.14144) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:50.622216+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:50.623018+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:50.628167+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:50.966191+0000 mgr.y (mgr.14144) 32 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi029=y;smithi103=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:17:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:17:50.966876+0000 mgr.y (mgr.14144) 33 : cephadm [INF] Saving service mgr spec with placement smithi029=y;smithi103=x;count:2 2024-09-17T09:17:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:50.970682+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:50.971350+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:50.976248+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:51.882 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:50.977241+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:17:51.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:50.979806+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T09:17:51.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:50.980558+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:17:51.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:50.981493+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:51.883 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:17:50.982177+0000 mgr.y (mgr.14144) 34 : cephadm [INF] Deploying daemon mgr.x on smithi103 2024-09-17T09:17:51.883 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:50.541529+0000 mgr.y (mgr.14144) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:51.883 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:50.622216+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:51.883 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:50.623018+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:51.883 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:50.628167+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:51.883 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:50.966191+0000 mgr.y (mgr.14144) 32 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi029=y;smithi103=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:17:51.883 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:17:50.966876+0000 mgr.y (mgr.14144) 33 : cephadm [INF] Saving service mgr spec with placement smithi029=y;smithi103=x;count:2 2024-09-17T09:17:51.883 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:50.970682+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:51.883 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:50.971350+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:51.884 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:50.976248+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:51.884 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:50.977241+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:17:51.884 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:50.979806+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T09:17:51.884 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:50.980558+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:17:51.884 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:50.981493+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:51.884 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:17:50.982177+0000 mgr.y (mgr.14144) 34 : cephadm [INF] Deploying daemon mgr.x on smithi103 2024-09-17T09:17:51.884 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-09-17T09:17:51.885 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-09-17T09:17:51.885 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000118257 s, 4.3 MB/s 2024-09-17T09:17:51.886 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T09:17:51.943 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-09-17T09:17:52.000 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T09:17:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:17:52.001 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-17T09:17:52.001 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:17:52.001 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:17:52.001 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-09-17 09:17:26.246381605 +0000 2024-09-17T09:17:52.001 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-09-17 09:15:21.645328821 +0000 2024-09-17T09:17:52.001 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-09-17 09:15:21.645328821 +0000 2024-09-17T09:17:52.001 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-09-17 09:15:21.645328821 +0000 2024-09-17T09:17:52.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T09:17:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-09-17T09:17:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-09-17T09:17:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000122578 s, 4.2 MB/s 2024-09-17T09:17:52.068 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T09:17:52.125 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-09-17T09:17:52.182 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T09:17:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:17:52.183 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-17T09:17:52.183 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:17:52.183 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:17:52.183 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-09-17 09:17:26.247381589 +0000 2024-09-17T09:17:52.183 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-09-17 09:15:21.963323588 +0000 2024-09-17T09:17:52.183 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-09-17 09:15:21.963323588 +0000 2024-09-17T09:17:52.183 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-09-17 09:15:21.963323588 +0000 2024-09-17T09:17:52.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T09:17:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-09-17T09:17:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-09-17T09:17:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.1946e-05 s, 5.6 MB/s 2024-09-17T09:17:52.249 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T09:17:52.306 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-17T09:17:52.306 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T09:17:52.322 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T09:17:52.322 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-09-17T09:17:52.377 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T09:17:52.378 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:17:52.378 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-17T09:17:52.378 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:17:52.378 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:17:52.378 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-17 09:17:36.914008610 +0000 2024-09-17T09:17:52.378 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-17 09:15:13.067767255 +0000 2024-09-17T09:17:52.378 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-17 09:15:13.067767255 +0000 2024-09-17T09:17:52.378 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-17 09:15:13.067767255 +0000 2024-09-17T09:17:52.378 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T09:17:52.441 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-17T09:17:52.441 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-17T09:17:52.442 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000207939 s, 2.5 MB/s 2024-09-17T09:17:52.443 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T09:17:52.500 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-09-17T09:17:52.555 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T09:17:52.555 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:17:52.555 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-17T09:17:52.556 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:17:52.556 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:17:52.556 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-17 09:17:36.914008610 +0000 2024-09-17T09:17:52.556 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-17 09:15:13.390761226 +0000 2024-09-17T09:17:52.556 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-17 09:15:13.390761226 +0000 2024-09-17T09:17:52.556 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-17 09:15:13.390761226 +0000 2024-09-17T09:17:52.556 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T09:17:52.738 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-17T09:17:52.738 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-17T09:17:52.738 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000217803 s, 2.4 MB/s 2024-09-17T09:17:52.739 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T09:17:52.756 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-09-17T09:17:52.817 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T09:17:52.817 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:17:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-17T09:17:52.817 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:17:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:17:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-17 09:17:52.694710974 +0000 2024-09-17T09:17:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-17 09:15:13.719755085 +0000 2024-09-17T09:17:52.818 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-17 09:15:13.719755085 +0000 2024-09-17T09:17:52.818 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-17 09:15:13.719755085 +0000 2024-09-17T09:17:52.818 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T09:17:52.915 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-17T09:17:52.916 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-17T09:17:52.916 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000214421 s, 2.4 MB/s 2024-09-17T09:17:52.917 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T09:17:52.932 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-09-17T09:17:52.989 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T09:17:52.990 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T09:17:52.990 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-17T09:17:52.990 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T09:17:52.990 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T09:17:52.990 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-09-17 09:17:52.874707579 +0000 2024-09-17T09:17:52.990 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-09-17 09:15:14.076748422 +0000 2024-09-17T09:17:52.990 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-09-17 09:15:14.076748422 +0000 2024-09-17T09:17:52.990 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-09-17 09:15:14.076748422 +0000 2024-09-17T09:17:52.991 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T09:17:53.053 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-09-17T09:17:53.053 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-09-17T09:17:53.053 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000177621 s, 2.9 MB/s 2024-09-17T09:17:53.054 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T09:17:53.111 INFO:tasks.cephadm:Deploying osd.0 on smithi029 with /dev/vg_nvme/lv_4... 2024-09-17T09:17:53.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-17T09:17:53.210 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:17:53 smithi103 podman[24750]: 2024-09-17 09:17:53.113852291 +0000 UTC m=+0.123227230 container create 1a97fa2150f2908fc5d8643f9e897435075aa45292ab3abca9bd0cf0d78cff1c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS) 2024-09-17T09:17:53.505 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:17:53 smithi103 podman[24750]: 2024-09-17 09:17:53.315929459 +0000 UTC m=+0.325304405 container init 1a97fa2150f2908fc5d8643f9e897435075aa45292ab3abca9bd0cf0d78cff1c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0) 2024-09-17T09:17:53.505 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:17:53 smithi103 podman[24750]: 2024-09-17 09:17:53.318577453 +0000 UTC m=+0.327952400 container start 1a97fa2150f2908fc5d8643f9e897435075aa45292ab3abca9bd0cf0d78cff1c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x, GIT_CLEAN=True, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, RELEASE=HEAD, ceph=True) 2024-09-17T09:17:53.506 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:17:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[24804]: debug 2024-09-17T09:17:53.336+0000 7f789ed57700 1 -- 172.21.15.103:0/2529136523 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e26e77c4e0 con 0x55e26e750c00 2024-09-17T09:17:53.506 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:17:53 smithi103 bash[24750]: 1a97fa2150f2908fc5d8643f9e897435075aa45292ab3abca9bd0cf0d78cff1c 2024-09-17T09:17:53.506 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:17:53 smithi103 systemd[1]: Started Ceph mgr.x for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:17:54.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:52.541910+0000 mgr.y (mgr.14144) 35 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:54.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:53.485188+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:54.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:53.486106+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:52.541910+0000 mgr.y (mgr.14144) 35 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:53.485188+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:53.486106+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:54.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:52.541910+0000 mgr.y (mgr.14144) 35 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:54.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:53.485188+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:54.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:53.486106+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:55.078 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-17T09:17:55.078 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-17T09:17:55.078 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-09-17T09:17:55.078 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-09-17T09:17:55.078 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00832767 s, 1.3 GB/s 2024-09-17T09:17:55.078 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-09-17T09:17:55.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:54.399465+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:55.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:54.404620+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:55.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:17:54.405506+0000 mgr.y (mgr.14144) 36 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-17T09:17:55.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:54.405777+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:17:55.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:54.406533+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:17:55.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:54.407139+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:55.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:17:54.407706+0000 mgr.y (mgr.14144) 37 : cephadm [INF] Reconfiguring daemon mgr.y on smithi029 2024-09-17T09:17:55.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:54.542211+0000 mgr.y (mgr.14144) 38 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:55.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:54.399465+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:55.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:54.404620+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:55.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:17:54.405506+0000 mgr.y (mgr.14144) 36 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-17T09:17:55.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:54.405777+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:17:55.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:54.406533+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:17:55.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:54.407139+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:55.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:17:54.407706+0000 mgr.y (mgr.14144) 37 : cephadm [INF] Reconfiguring daemon mgr.y on smithi029 2024-09-17T09:17:55.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:54.542211+0000 mgr.y (mgr.14144) 38 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:55.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:54.399465+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:55.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:54.404620+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:17:54.405506+0000 mgr.y (mgr.14144) 36 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-17T09:17:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:54.405777+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:17:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:54.406533+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:17:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:54.407139+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:17:54.407706+0000 mgr.y (mgr.14144) 37 : cephadm [INF] Reconfiguring daemon mgr.y on smithi029 2024-09-17T09:17:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:54.542211+0000 mgr.y (mgr.14144) 38 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:56.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_4 2024-09-17T09:17:57.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:56.107105+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:57.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:56.107867+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:57.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:56.112323+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:57.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:56.107105+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:57.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:56.107867+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:57.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:56.112323+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:57.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:56.107105+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:57.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:56.107867+0000 mon.a (mon.0) 198 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:17:57.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:56.112323+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:17:58.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:56.542469+0000 mgr.y (mgr.14144) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:58.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:56.542469+0000 mgr.y (mgr.14144) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:58.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:56.542469+0000 mgr.y (mgr.14144) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:17:59.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:58.223546+0000 mon.a (mon.0) 200 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:17:59.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:58.224198+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.103:0/1724448992' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:17:59.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:58.224553+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.103:0/1724448992' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:17:59.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:58.225350+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.103:0/1724448992' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:17:59.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:58.225628+0000 mon.b (mon.2) 6 : audit [DBG] from='mgr.? 172.21.15.103:0/1724448992' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:17:59.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:58.401174+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:17:59.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:58.402684+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:17:59.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:58.403220+0000 mon.a (mon.0) 203 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:59.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:58.223546+0000 mon.a (mon.0) 200 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:17:59.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:58.224198+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.103:0/1724448992' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:17:59.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:58.224553+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.103:0/1724448992' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:17:59.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:58.225350+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.103:0/1724448992' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:17:59.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:58.225628+0000 mon.b (mon.2) 6 : audit [DBG] from='mgr.? 172.21.15.103:0/1724448992' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:17:59.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:58.401174+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:17:59.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:58.402684+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:17:59.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:17:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:58.403220+0000 mon.a (mon.0) 203 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:17:59.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:58.223546+0000 mon.a (mon.0) 200 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:17:59.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:58.224198+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.103:0/1724448992' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:17:59.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:58.224553+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.103:0/1724448992' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:17:59.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:58.225350+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.103:0/1724448992' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:17:59.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:58.225628+0000 mon.b (mon.2) 6 : audit [DBG] from='mgr.? 172.21.15.103:0/1724448992' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:17:59.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:58.401174+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:17:59.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:58.402684+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:17:59.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:17:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:58.403220+0000 mon.a (mon.0) 203 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:58.400301+0000 mgr.y (mgr.14144) 40 : audit [DBG] from='client.14202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:18:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:17:58.402458+0000 mgr.y (mgr.14144) 41 : cephadm [INF] Applying service osd.None on host smithi029... 2024-09-17T09:18:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:58.542708+0000 mgr.y (mgr.14144) 42 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:17:59.123770+0000 mon.a (mon.0) 204 : cluster [DBG] mgrmap e14: y(active, since 54s), standbys: x 2024-09-17T09:18:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:17:59.123894+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:18:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:58.400301+0000 mgr.y (mgr.14144) 40 : audit [DBG] from='client.14202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:18:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:17:58.402458+0000 mgr.y (mgr.14144) 41 : cephadm [INF] Applying service osd.None on host smithi029... 2024-09-17T09:18:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:58.542708+0000 mgr.y (mgr.14144) 42 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:17:59.123770+0000 mon.a (mon.0) 204 : cluster [DBG] mgrmap e14: y(active, since 54s), standbys: x 2024-09-17T09:18:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:17:59.123894+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:18:00.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:58.400301+0000 mgr.y (mgr.14144) 40 : audit [DBG] from='client.14202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:18:00.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:17:58.402458+0000 mgr.y (mgr.14144) 41 : cephadm [INF] Applying service osd.None on host smithi029... 2024-09-17T09:18:00.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:58.542708+0000 mgr.y (mgr.14144) 42 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:17:59.123770+0000 mon.a (mon.0) 204 : cluster [DBG] mgrmap e14: y(active, since 54s), standbys: x 2024-09-17T09:18:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:17:59.123894+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:18:01.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:00.542424+0000 mon.a (mon.0) 206 : audit [INF] from='client.? 172.21.15.29:0/2005698034' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97f6429c-ffd4-4aaf-bcf3-61a379d19953"}]: dispatch 2024-09-17T09:18:01.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:00.545822+0000 mon.a (mon.0) 207 : audit [INF] from='client.? 172.21.15.29:0/2005698034' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97f6429c-ffd4-4aaf-bcf3-61a379d19953"}]': finished 2024-09-17T09:18:01.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:00.545878+0000 mon.a (mon.0) 208 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-17T09:18:01.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:00.545976+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:01.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:00.989580+0000 mon.a (mon.0) 210 : audit [DBG] from='client.? 172.21.15.29:0/1356102273' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:18:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:00.542424+0000 mon.a (mon.0) 206 : audit [INF] from='client.? 172.21.15.29:0/2005698034' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97f6429c-ffd4-4aaf-bcf3-61a379d19953"}]: dispatch 2024-09-17T09:18:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:00.545822+0000 mon.a (mon.0) 207 : audit [INF] from='client.? 172.21.15.29:0/2005698034' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97f6429c-ffd4-4aaf-bcf3-61a379d19953"}]': finished 2024-09-17T09:18:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:00.545878+0000 mon.a (mon.0) 208 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-17T09:18:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:00.545976+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:00.989580+0000 mon.a (mon.0) 210 : audit [DBG] from='client.? 172.21.15.29:0/1356102273' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:18:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:00.542424+0000 mon.a (mon.0) 206 : audit [INF] from='client.? 172.21.15.29:0/2005698034' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97f6429c-ffd4-4aaf-bcf3-61a379d19953"}]: dispatch 2024-09-17T09:18:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:00.545822+0000 mon.a (mon.0) 207 : audit [INF] from='client.? 172.21.15.29:0/2005698034' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97f6429c-ffd4-4aaf-bcf3-61a379d19953"}]': finished 2024-09-17T09:18:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:00.545878+0000 mon.a (mon.0) 208 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-17T09:18:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:00.545976+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:00.989580+0000 mon.a (mon.0) 210 : audit [DBG] from='client.? 172.21.15.29:0/1356102273' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:18:02.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:00.542927+0000 mgr.y (mgr.14144) 43 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:02.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:00.542927+0000 mgr.y (mgr.14144) 43 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:02.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:00.542927+0000 mgr.y (mgr.14144) 43 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:04.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:02.543287+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:04.267 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:02.543287+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:04.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:02.543287+0000 mgr.y (mgr.14144) 44 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:05.187 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:04.667773+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:18:05.187 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:04.668731+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:18:05.188 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:04.667773+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:18:05.188 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:04.668731+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:18:05.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:04.667773+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:18:05.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:04.668731+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:18:06.426 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:04.543528+0000 mgr.y (mgr.14144) 45 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:06.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:04.543528+0000 mgr.y (mgr.14144) 45 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:06.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:04.543528+0000 mgr.y (mgr.14144) 45 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:07.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:06.449902+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T09:18:07.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:06.450623+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:07.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:06.449902+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T09:18:07.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:06.450623+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:07.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:06.449902+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T09:18:07.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:06.450623+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:08.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:18:06.451190+0000 mgr.y (mgr.14144) 46 : cephadm [INF] Deploying daemon osd.0 on smithi029 2024-09-17T09:18:08.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:06.543771+0000 mgr.y (mgr.14144) 47 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:08.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:18:06.451190+0000 mgr.y (mgr.14144) 46 : cephadm [INF] Deploying daemon osd.0 on smithi029 2024-09-17T09:18:08.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:06.543771+0000 mgr.y (mgr.14144) 47 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:08.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:18:06.451190+0000 mgr.y (mgr.14144) 46 : cephadm [INF] Deploying daemon osd.0 on smithi029 2024-09-17T09:18:08.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:06.543771+0000 mgr.y (mgr.14144) 47 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:10.049 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 0 on host 'smithi029' 2024-09-17T09:18:10.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:08.544016+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:10.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:10.038067+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:10.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:10.041538+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:10.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:10.045598+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:10.352 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:08.544016+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:10.352 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:10.038067+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:10.352 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:10.041538+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:10.352 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:10.045598+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:10.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:08.544016+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:10.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:10.038067+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:10.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:10.041538+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:10.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:10.045598+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:10.661 DEBUG:teuthology.orchestra.run.smithi029:osd.0> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.0.service 2024-09-17T09:18:10.663 INFO:tasks.cephadm:Deploying osd.1 on smithi029 with /dev/vg_nvme/lv_3... 2024-09-17T09:18:10.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-17T09:18:12.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:10.544224+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:12.235 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:10.544224+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:12.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:10.544224+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:13.133 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-17T09:18:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-17T09:18:13.134 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-09-17T09:18:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-09-17T09:18:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125708 s, 834 MB/s 2024-09-17T09:18:13.134 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-09-17T09:18:13.535 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:18:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.0[29244]: debug 2024-09-17T09:18:13.134+0000 7fbb97f1f080 -1 osd.0 0 log_to_monitors {default=true} 2024-09-17T09:18:14.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:12.544501+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:14.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:12.936469+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:14.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:13.138761+0000 mon.c (mon.1) 4 : audit [INF] from='osd.0 [v2:172.21.15.29:6802/3135009490,v1:172.21.15.29:6803/3135009490]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:18:14.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:13.139143+0000 mon.a (mon.0) 219 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:18:14.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:12.544501+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:14.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:12.936469+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:14.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:13.138761+0000 mon.c (mon.1) 4 : audit [INF] from='osd.0 [v2:172.21.15.29:6802/3135009490,v1:172.21.15.29:6803/3135009490]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:18:14.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:13.139143+0000 mon.a (mon.0) 219 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:18:14.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:12.544501+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:14.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:12.936469+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:14.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:13.138761+0000 mon.c (mon.1) 4 : audit [INF] from='osd.0 [v2:172.21.15.29:6802/3135009490,v1:172.21.15.29:6803/3135009490]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:18:14.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:13.139143+0000 mon.a (mon.0) 219 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:18:15.039 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_3 2024-09-17T09:18:15.239 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:18:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.0[29244]: debug 2024-09-17T09:18:14.978+0000 7fbb8e490700 -1 osd.0 0 waiting for initial osdmap 2024-09-17T09:18:15.239 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:18:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.0[29244]: debug 2024-09-17T09:18:14.981+0000 7fbb89409700 -1 osd.0 7 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2024-09-17T09:18:15.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:13.971662+0000 mon.a (mon.0) 220 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T09:18:15.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:13.971746+0000 mon.a (mon.0) 221 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-17T09:18:15.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:13.971910+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:15.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:13.972404+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.29:6802/3135009490,v1:172.21.15.29:6803/3135009490]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:15.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:13.972758+0000 mon.a (mon.0) 223 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:15.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:13.971662+0000 mon.a (mon.0) 220 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T09:18:15.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:13.971746+0000 mon.a (mon.0) 221 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-17T09:18:15.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:13.971910+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:15.241 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:13.972404+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.29:6802/3135009490,v1:172.21.15.29:6803/3135009490]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:15.241 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:13.972758+0000 mon.a (mon.0) 223 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:15.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:13.971662+0000 mon.a (mon.0) 220 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T09:18:15.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:13.971746+0000 mon.a (mon.0) 221 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-17T09:18:15.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:13.971910+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:15.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:13.972404+0000 mon.c (mon.1) 5 : audit [INF] from='osd.0 [v2:172.21.15.29:6802/3135009490,v1:172.21.15.29:6803/3135009490]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:15.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:13.972758+0000 mon.a (mon.0) 223 : audit [INF] from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:16.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:14.544839+0000 mgr.y (mgr.14144) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:16.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:14.973727+0000 mon.a (mon.0) 224 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-17T09:18:16.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:14.973847+0000 mon.a (mon.0) 225 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-17T09:18:16.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:14.974303+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:16.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:14.979391+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:16.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:14.544839+0000 mgr.y (mgr.14144) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:16.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:14.973727+0000 mon.a (mon.0) 224 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-17T09:18:16.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:14.973847+0000 mon.a (mon.0) 225 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-17T09:18:16.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:14.974303+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:16.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:14.979391+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:16.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:14.544839+0000 mgr.y (mgr.14144) 51 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:16.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:14.973727+0000 mon.a (mon.0) 224 : audit [INF] from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-17T09:18:16.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:14.973847+0000 mon.a (mon.0) 225 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-17T09:18:16.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:14.974303+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:16.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:14.979391+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:17.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:14.165620+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:18:17.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:14.165847+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:18:17.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:15.979326+0000 mon.a (mon.0) 228 : cluster [INF] osd.0 [v2:172.21.15.29:6802/3135009490,v1:172.21.15.29:6803/3135009490] boot 2024-09-17T09:18:17.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:15.979369+0000 mon.a (mon.0) 229 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-17T09:18:17.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:15.979711+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:17.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:14.165620+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:18:17.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:14.165847+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:18:17.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:15.979326+0000 mon.a (mon.0) 228 : cluster [INF] osd.0 [v2:172.21.15.29:6802/3135009490,v1:172.21.15.29:6803/3135009490] boot 2024-09-17T09:18:17.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:15.979369+0000 mon.a (mon.0) 229 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-17T09:18:17.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:15.979711+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:17.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:14.165620+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:18:17.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:14.165847+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:18:17.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:15.979326+0000 mon.a (mon.0) 228 : cluster [INF] osd.0 [v2:172.21.15.29:6802/3135009490,v1:172.21.15.29:6803/3135009490] boot 2024-09-17T09:18:17.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:15.979369+0000 mon.a (mon.0) 229 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-17T09:18:17.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:15.979711+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:18:18.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:16.545114+0000 mgr.y (mgr.14144) 52 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:18.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:16.988312+0000 mon.a (mon.0) 231 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-17T09:18:18.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:18:17.227825+0000 mgr.y (mgr.14144) 53 : cephadm [INF] Detected new or changed devices on smithi029 2024-09-17T09:18:18.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:17.232773+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:18.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:17.470875+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:18.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:17.472087+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:18.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:17.476874+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:18.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:17.651833+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:18:18.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:17.653176+0000 mon.a (mon.0) 237 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:18:18.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:17.653738+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:18.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:16.545114+0000 mgr.y (mgr.14144) 52 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:18.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:16.988312+0000 mon.a (mon.0) 231 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-17T09:18:18.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:18:17.227825+0000 mgr.y (mgr.14144) 53 : cephadm [INF] Detected new or changed devices on smithi029 2024-09-17T09:18:18.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:17.232773+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:18.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:17.470875+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:18.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:17.472087+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:18.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:17.476874+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:18.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:17.651833+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:18:18.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:17.653176+0000 mon.a (mon.0) 237 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:18:18.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:17.653738+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:18.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:16.545114+0000 mgr.y (mgr.14144) 52 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T09:18:18.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:16.988312+0000 mon.a (mon.0) 231 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-17T09:18:18.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:18:17.227825+0000 mgr.y (mgr.14144) 53 : cephadm [INF] Detected new or changed devices on smithi029 2024-09-17T09:18:18.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:17.232773+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:18.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:17.470875+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:18.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:17.472087+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:18.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:17.476874+0000 mon.a (mon.0) 235 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:18.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:17.651833+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:18:18.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:17.653176+0000 mon.a (mon.0) 237 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:18:18.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:17.653738+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:19.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:17.651110+0000 mgr.y (mgr.14144) 54 : audit [DBG] from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:18:19.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:18:17.652971+0000 mgr.y (mgr.14144) 55 : cephadm [INF] Applying service osd.None on host smithi029... 2024-09-17T09:18:19.152 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:17.651110+0000 mgr.y (mgr.14144) 54 : audit [DBG] from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:18:19.152 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:18:17.652971+0000 mgr.y (mgr.14144) 55 : cephadm [INF] Applying service osd.None on host smithi029... 2024-09-17T09:18:19.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:17.651110+0000 mgr.y (mgr.14144) 54 : audit [DBG] from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:18:19.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:18:17.652971+0000 mgr.y (mgr.14144) 55 : cephadm [INF] Applying service osd.None on host smithi029... 2024-09-17T09:18:20.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:18.545354+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:20.309 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:19.887070+0000 mon.a (mon.0) 239 : audit [INF] from='client.? 172.21.15.29:0/492956080' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec88dcae-85cf-4f6c-8f06-c2d891eaf2c3"}]: dispatch 2024-09-17T09:18:20.309 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:19.890474+0000 mon.a (mon.0) 240 : audit [INF] from='client.? 172.21.15.29:0/492956080' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec88dcae-85cf-4f6c-8f06-c2d891eaf2c3"}]': finished 2024-09-17T09:18:20.309 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:19.890545+0000 mon.a (mon.0) 241 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-17T09:18:20.309 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:19.890652+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:20.310 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:18.545354+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:20.310 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:19.887070+0000 mon.a (mon.0) 239 : audit [INF] from='client.? 172.21.15.29:0/492956080' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec88dcae-85cf-4f6c-8f06-c2d891eaf2c3"}]: dispatch 2024-09-17T09:18:20.310 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:19.890474+0000 mon.a (mon.0) 240 : audit [INF] from='client.? 172.21.15.29:0/492956080' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec88dcae-85cf-4f6c-8f06-c2d891eaf2c3"}]': finished 2024-09-17T09:18:20.310 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:19.890545+0000 mon.a (mon.0) 241 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-17T09:18:20.310 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:19.890652+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:20.311 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:18.545354+0000 mgr.y (mgr.14144) 56 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:20.311 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:19.887070+0000 mon.a (mon.0) 239 : audit [INF] from='client.? 172.21.15.29:0/492956080' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec88dcae-85cf-4f6c-8f06-c2d891eaf2c3"}]: dispatch 2024-09-17T09:18:20.311 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:19.890474+0000 mon.a (mon.0) 240 : audit [INF] from='client.? 172.21.15.29:0/492956080' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec88dcae-85cf-4f6c-8f06-c2d891eaf2c3"}]': finished 2024-09-17T09:18:20.311 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:19.890545+0000 mon.a (mon.0) 241 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-17T09:18:20.311 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:19.890652+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:21.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:20.321895+0000 mon.c (mon.1) 6 : audit [DBG] from='client.? 172.21.15.29:0/1978089471' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:18:21.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:20.321895+0000 mon.c (mon.1) 6 : audit [DBG] from='client.? 172.21.15.29:0/1978089471' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:18:21.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:20.321895+0000 mon.c (mon.1) 6 : audit [DBG] from='client.? 172.21.15.29:0/1978089471' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:18:22.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:20.545853+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:22.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:20.545853+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:22.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:20.545853+0000 mgr.y (mgr.14144) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:24.109 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:22.546180+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:24.109 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:22.546180+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:24.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:22.546180+0000 mgr.y (mgr.14144) 58 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:26.186 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:24.546565+0000 mgr.y (mgr.14144) 59 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:26.187 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:25.868438+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T09:18:26.187 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:25.869197+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:26.187 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:24.546565+0000 mgr.y (mgr.14144) 59 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:26.187 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:25.868438+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T09:18:26.187 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:25.869197+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:26.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:24.546565+0000 mgr.y (mgr.14144) 59 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:26.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:25.868438+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T09:18:26.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:25.869197+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:27.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:18:25.869819+0000 mgr.y (mgr.14144) 60 : cephadm [INF] Deploying daemon osd.1 on smithi029 2024-09-17T09:18:27.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:18:25.869819+0000 mgr.y (mgr.14144) 60 : cephadm [INF] Deploying daemon osd.1 on smithi029 2024-09-17T09:18:27.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:18:25.869819+0000 mgr.y (mgr.14144) 60 : cephadm [INF] Deploying daemon osd.1 on smithi029 2024-09-17T09:18:28.278 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:26.546997+0000 mgr.y (mgr.14144) 61 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:28.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:26.546997+0000 mgr.y (mgr.14144) 61 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:28.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:26.546997+0000 mgr.y (mgr.14144) 61 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:29.589 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 1 on host 'smithi029' 2024-09-17T09:18:30.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:28.547415+0000 mgr.y (mgr.14144) 62 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:30.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:29.575458+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:30.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:29.580084+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:30.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:29.585982+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:30.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:28.547415+0000 mgr.y (mgr.14144) 62 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:30.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:29.575458+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:30.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:29.580084+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:30.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:29.585982+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:30.274 DEBUG:teuthology.orchestra.run.smithi029:osd.1> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.1.service 2024-09-17T09:18:30.277 INFO:tasks.cephadm:Deploying osd.2 on smithi029 with /dev/vg_nvme/lv_2... 2024-09-17T09:18:30.277 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-17T09:18:30.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:28.547415+0000 mgr.y (mgr.14144) 62 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:30.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:29.575458+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:30.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:29.580084+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:30.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:29.585982+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:32.265 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:30.547749+0000 mgr.y (mgr.14144) 63 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:32.265 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:30.547749+0000 mgr.y (mgr.14144) 63 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:32.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:30.547749+0000 mgr.y (mgr.14144) 63 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:32.761 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:18:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.1[31661]: debug 2024-09-17T09:18:32.519+0000 7f42283d3080 -1 osd.1 0 log_to_monitors {default=true} 2024-09-17T09:18:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-17T09:18:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-17T09:18:32.762 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-09-17T09:18:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-09-17T09:18:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0143117 s, 733 MB/s 2024-09-17T09:18:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-09-17T09:18:33.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:32.523619+0000 mon.a (mon.0) 248 : audit [INF] from='osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:18:33.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:32.587055+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:33.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:32.523619+0000 mon.a (mon.0) 248 : audit [INF] from='osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:18:33.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:32.587055+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:33.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:32.523619+0000 mon.a (mon.0) 248 : audit [INF] from='osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:18:33.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:32.587055+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:34.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:18:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.1[31661]: debug 2024-09-17T09:18:34.026+0000 7f421e944700 -1 osd.1 0 waiting for initial osdmap 2024-09-17T09:18:34.236 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:18:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.1[31661]: debug 2024-09-17T09:18:34.030+0000 7f42190bc700 -1 osd.1 12 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2024-09-17T09:18:34.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:32.548058+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:34.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:33.020175+0000 mon.a (mon.0) 250 : audit [INF] from='osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T09:18:34.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:33.020256+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-17T09:18:34.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:33.020369+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:34.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:33.020753+0000 mon.a (mon.0) 253 : audit [INF] from='osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:34.237 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:32.548058+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:34.237 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:33.020175+0000 mon.a (mon.0) 250 : audit [INF] from='osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T09:18:34.237 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:33.020256+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-17T09:18:34.237 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:33.020369+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:34.237 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:33.020753+0000 mon.a (mon.0) 253 : audit [INF] from='osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:34.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:32.548058+0000 mgr.y (mgr.14144) 64 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:34.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:33.020175+0000 mon.a (mon.0) 250 : audit [INF] from='osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T09:18:34.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:33.020256+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-17T09:18:34.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:33.020369+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:34.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:33.020753+0000 mon.a (mon.0) 253 : audit [INF] from='osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:34.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_2 2024-09-17T09:18:35.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:34.021991+0000 mon.a (mon.0) 254 : audit [INF] from='osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-17T09:18:35.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:34.022070+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-17T09:18:35.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:34.022516+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:35.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:34.027113+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:35.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:34.021991+0000 mon.a (mon.0) 254 : audit [INF] from='osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-17T09:18:35.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:34.022070+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-17T09:18:35.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:34.022516+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:35.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:34.027113+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:35.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:34.021991+0000 mon.a (mon.0) 254 : audit [INF] from='osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-17T09:18:35.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:34.022070+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-17T09:18:35.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:34.022516+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:35.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:34.027113+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:33.512376+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:18:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:33.512547+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:18:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:34.548343+0000 mgr.y (mgr.14144) 65 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:35.026838+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:35.031232+0000 mon.a (mon.0) 259 : cluster [INF] osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162] boot 2024-09-17T09:18:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:35.031265+0000 mon.a (mon.0) 260 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-17T09:18:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:35.031666+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:36.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:33.512376+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:18:36.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:33.512547+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:18:36.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:34.548343+0000 mgr.y (mgr.14144) 65 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:36.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:35.026838+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:36.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:35.031232+0000 mon.a (mon.0) 259 : cluster [INF] osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162] boot 2024-09-17T09:18:36.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:35.031265+0000 mon.a (mon.0) 260 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-17T09:18:36.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:35.031666+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:36.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:33.512376+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:18:36.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:33.512547+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:18:36.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:34.548343+0000 mgr.y (mgr.14144) 65 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-09-17T09:18:36.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:35.026838+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:36.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:35.031232+0000 mon.a (mon.0) 259 : cluster [INF] osd.1 [v2:172.21.15.29:6810/1986066162,v1:172.21.15.29:6811/1986066162] boot 2024-09-17T09:18:36.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:35.031265+0000 mon.a (mon.0) 260 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-17T09:18:36.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:35.031666+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:18:37.066 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:36.036444+0000 mon.a (mon.0) 262 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-17T09:18:37.066 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:36.617988+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:37.066 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:36.620971+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:37.067 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:36.622077+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:37.067 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:36.626722+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:37.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:36.036444+0000 mon.a (mon.0) 262 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-17T09:18:37.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:36.617988+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:37.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:36.620971+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:37.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:36.622077+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:37.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:36.626722+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:37.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:36.036444+0000 mon.a (mon.0) 262 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-17T09:18:37.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:36.617988+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:37.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:36.620971+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:37.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:36.622077+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:37.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:36.626722+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:38.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:36.548607+0000 mgr.y (mgr.14144) 66 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:38.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:18:36.613891+0000 mgr.y (mgr.14144) 67 : cephadm [INF] Detected new or changed devices on smithi029 2024-09-17T09:18:38.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:37.480015+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:18:38.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:37.481339+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:18:38.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:37.481913+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:38.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:36.548607+0000 mgr.y (mgr.14144) 66 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:38.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:18:36.613891+0000 mgr.y (mgr.14144) 67 : cephadm [INF] Detected new or changed devices on smithi029 2024-09-17T09:18:38.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:37.480015+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:18:38.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:37.481339+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:18:38.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:37.481913+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:38.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:36.548607+0000 mgr.y (mgr.14144) 66 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:38.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:18:36.613891+0000 mgr.y (mgr.14144) 67 : cephadm [INF] Detected new or changed devices on smithi029 2024-09-17T09:18:38.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:37.480015+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:18:38.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:37.481339+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:18:38.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:37.481913+0000 mon.a (mon.0) 269 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:39.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:37.479338+0000 mgr.y (mgr.14144) 68 : audit [DBG] from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:18:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:18:37.481092+0000 mgr.y (mgr.14144) 69 : cephadm [INF] Applying service osd.None on host smithi029... 2024-09-17T09:18:39.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:37.479338+0000 mgr.y (mgr.14144) 68 : audit [DBG] from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:18:39.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:18:37.481092+0000 mgr.y (mgr.14144) 69 : cephadm [INF] Applying service osd.None on host smithi029... 2024-09-17T09:18:39.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:37.479338+0000 mgr.y (mgr.14144) 68 : audit [DBG] from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:18:39.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:18:37.481092+0000 mgr.y (mgr.14144) 69 : cephadm [INF] Applying service osd.None on host smithi029... 2024-09-17T09:18:40.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:38.549022+0000 mgr.y (mgr.14144) 70 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:40.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:39.687973+0000 mon.c (mon.1) 7 : audit [INF] from='client.? 172.21.15.29:0/1959746390' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfb3bfde-d7f5-4570-b530-405f354cbb9a"}]: dispatch 2024-09-17T09:18:40.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:39.688328+0000 mon.a (mon.0) 270 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfb3bfde-d7f5-4570-b530-405f354cbb9a"}]: dispatch 2024-09-17T09:18:40.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:39.692216+0000 mon.a (mon.0) 271 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dfb3bfde-d7f5-4570-b530-405f354cbb9a"}]': finished 2024-09-17T09:18:40.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:39.692299+0000 mon.a (mon.0) 272 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-09-17T09:18:40.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:39.692397+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:40.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:38.549022+0000 mgr.y (mgr.14144) 70 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:40.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:39.687973+0000 mon.c (mon.1) 7 : audit [INF] from='client.? 172.21.15.29:0/1959746390' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfb3bfde-d7f5-4570-b530-405f354cbb9a"}]: dispatch 2024-09-17T09:18:40.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:39.688328+0000 mon.a (mon.0) 270 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfb3bfde-d7f5-4570-b530-405f354cbb9a"}]: dispatch 2024-09-17T09:18:40.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:39.692216+0000 mon.a (mon.0) 271 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dfb3bfde-d7f5-4570-b530-405f354cbb9a"}]': finished 2024-09-17T09:18:40.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:39.692299+0000 mon.a (mon.0) 272 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-09-17T09:18:40.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:39.692397+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:40.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:38.549022+0000 mgr.y (mgr.14144) 70 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:40.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:39.687973+0000 mon.c (mon.1) 7 : audit [INF] from='client.? 172.21.15.29:0/1959746390' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfb3bfde-d7f5-4570-b530-405f354cbb9a"}]: dispatch 2024-09-17T09:18:40.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:39.688328+0000 mon.a (mon.0) 270 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfb3bfde-d7f5-4570-b530-405f354cbb9a"}]: dispatch 2024-09-17T09:18:40.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:39.692216+0000 mon.a (mon.0) 271 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dfb3bfde-d7f5-4570-b530-405f354cbb9a"}]': finished 2024-09-17T09:18:40.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:39.692299+0000 mon.a (mon.0) 272 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-09-17T09:18:40.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:39.692397+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:41.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:40.114169+0000 mon.a (mon.0) 274 : audit [DBG] from='client.? 172.21.15.29:0/1314088614' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:18:41.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:40.114169+0000 mon.a (mon.0) 274 : audit [DBG] from='client.? 172.21.15.29:0/1314088614' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:18:41.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:40.114169+0000 mon.a (mon.0) 274 : audit [DBG] from='client.? 172.21.15.29:0/1314088614' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:18:42.333 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:40.549428+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:42.333 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:40.549428+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:42.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:42 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:40.549428+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:44.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:42.549884+0000 mgr.y (mgr.14144) 72 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:44.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:42.549884+0000 mgr.y (mgr.14144) 72 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:44.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:42.549884+0000 mgr.y (mgr.14144) 72 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:46.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:44.550163+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:46.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:45.647410+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T09:18:46.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:45.648235+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:46.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:44.550163+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:46.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:45.647410+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T09:18:46.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:45.648235+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:46.467 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:44.550163+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:46.467 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:45.647410+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T09:18:46.467 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:45.648235+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:47.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:18:45.648880+0000 mgr.y (mgr.14144) 74 : cephadm [INF] Deploying daemon osd.2 on smithi029 2024-09-17T09:18:47.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:18:45.648880+0000 mgr.y (mgr.14144) 74 : cephadm [INF] Deploying daemon osd.2 on smithi029 2024-09-17T09:18:47.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:18:45.648880+0000 mgr.y (mgr.14144) 74 : cephadm [INF] Deploying daemon osd.2 on smithi029 2024-09-17T09:18:48.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:46.550469+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:48.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:46.550469+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:48.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:46.550469+0000 mgr.y (mgr.14144) 75 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:49.210 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 2 on host 'smithi029' 2024-09-17T09:18:49.894 DEBUG:teuthology.orchestra.run.smithi029:osd.2> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.2.service 2024-09-17T09:18:49.897 INFO:tasks.cephadm:Deploying osd.3 on smithi029 with /dev/vg_nvme/lv_1... 2024-09-17T09:18:49.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-17T09:18:50.219 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:48.550846+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:50.219 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:49.195632+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:50.219 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:49.200191+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:50.219 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:49.206677+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:50.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:48.550846+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:50.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:49.195632+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:50.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:49.200191+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:50.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:49.206677+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:50.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:48.550846+0000 mgr.y (mgr.14144) 76 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:50.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:49.195632+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:50.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:49.200191+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:50.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:49.206677+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:52.103 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:50.551171+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:52.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:50.551171+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:52.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:50.551171+0000 mgr.y (mgr.14144) 77 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:52.547 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:18:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.2[34048]: debug 2024-09-17T09:18:52.273+0000 7f796d0ee080 -1 osd.2 0 log_to_monitors {default=true} 2024-09-17T09:18:52.628 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-17T09:18:52.628 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-17T09:18:52.628 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-09-17T09:18:52.628 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-09-17T09:18:52.628 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00986074 s, 1.1 GB/s 2024-09-17T09:18:52.628 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-09-17T09:18:53.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:52.278503+0000 mon.c (mon.1) 8 : audit [INF] from='osd.2 [v2:172.21.15.29:6818/4166454918,v1:172.21.15.29:6819/4166454918]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:18:53.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:52.278879+0000 mon.a (mon.0) 280 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:18:53.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:52.400339+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:53.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:52.278503+0000 mon.c (mon.1) 8 : audit [INF] from='osd.2 [v2:172.21.15.29:6818/4166454918,v1:172.21.15.29:6819/4166454918]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:18:53.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:52.278879+0000 mon.a (mon.0) 280 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:18:53.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:52.400339+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:53.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi029:vg_nvme/lv_1 2024-09-17T09:18:53.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:52.278503+0000 mon.c (mon.1) 8 : audit [INF] from='osd.2 [v2:172.21.15.29:6818/4166454918,v1:172.21.15.29:6819/4166454918]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:18:53.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:52.278879+0000 mon.a (mon.0) 280 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:18:53.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:52.400339+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:54.266 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:52.551522+0000 mgr.y (mgr.14144) 78 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:54.266 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:53.078085+0000 mon.a (mon.0) 282 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T09:18:54.267 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:53.078173+0000 mon.a (mon.0) 283 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-17T09:18:54.267 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:53.078297+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:54.267 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:53.078736+0000 mon.c (mon.1) 9 : audit [INF] from='osd.2 [v2:172.21.15.29:6818/4166454918,v1:172.21.15.29:6819/4166454918]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:54.267 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:53.079026+0000 mon.a (mon.0) 285 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:54.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.2[34048]: debug 2024-09-17T09:18:54.087+0000 7f796365f700 -1 osd.2 0 waiting for initial osdmap 2024-09-17T09:18:54.267 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.2[34048]: debug 2024-09-17T09:18:54.091+0000 7f795ddd7700 -1 osd.2 17 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2024-09-17T09:18:54.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:52.551522+0000 mgr.y (mgr.14144) 78 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:54.268 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:53.078085+0000 mon.a (mon.0) 282 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T09:18:54.268 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:53.078173+0000 mon.a (mon.0) 283 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-17T09:18:54.268 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:53.078297+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:54.268 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:53.078736+0000 mon.c (mon.1) 9 : audit [INF] from='osd.2 [v2:172.21.15.29:6818/4166454918,v1:172.21.15.29:6819/4166454918]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:54.268 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:53.079026+0000 mon.a (mon.0) 285 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:54.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:52.551522+0000 mgr.y (mgr.14144) 78 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:54.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:53.078085+0000 mon.a (mon.0) 282 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T09:18:54.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:53.078173+0000 mon.a (mon.0) 283 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-17T09:18:54.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:53.078297+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:54.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:53.078736+0000 mon.c (mon.1) 9 : audit [INF] from='osd.2 [v2:172.21.15.29:6818/4166454918,v1:172.21.15.29:6819/4166454918]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:54.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:53.079026+0000 mon.a (mon.0) 285 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:18:55.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:54.080410+0000 mon.a (mon.0) 286 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-17T09:18:55.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:54.080526+0000 mon.a (mon.0) 287 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-17T09:18:55.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:54.080922+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:55.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:54.084428+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:55.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:55.084024+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:55.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:54.080410+0000 mon.a (mon.0) 286 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-17T09:18:55.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:54.080526+0000 mon.a (mon.0) 287 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-17T09:18:55.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:54.080922+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:55.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:54.084428+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:55.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:55.084024+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:55.982 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:54.080410+0000 mon.a (mon.0) 286 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-17T09:18:55.983 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:54.080526+0000 mon.a (mon.0) 287 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-17T09:18:55.983 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:54.080922+0000 mon.a (mon.0) 288 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:55.983 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:54.084428+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:55.983 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:55.084024+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:56.724 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:53.277911+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:18:56.724 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:53.278035+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:18:56.724 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:54.551870+0000 mgr.y (mgr.14144) 79 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:56.724 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:55.089064+0000 mon.a (mon.0) 291 : cluster [INF] osd.2 [v2:172.21.15.29:6818/4166454918,v1:172.21.15.29:6819/4166454918] boot 2024-09-17T09:18:56.725 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:55.089097+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-09-17T09:18:56.725 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:55.089427+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:56.725 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:55.089909+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2024-09-17T09:18:56.725 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:56.090625+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2024-09-17T09:18:56.725 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:56.090715+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-17T09:18:56.725 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:56.091487+0000 mon.a (mon.0) 297 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-09-17T09:18:56.725 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:53.277911+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:18:56.725 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:53.278035+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:18:56.725 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:54.551870+0000 mgr.y (mgr.14144) 79 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:56.725 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:55.089064+0000 mon.a (mon.0) 291 : cluster [INF] osd.2 [v2:172.21.15.29:6818/4166454918,v1:172.21.15.29:6819/4166454918] boot 2024-09-17T09:18:56.726 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:55.089097+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-09-17T09:18:56.726 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:55.089427+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:56.726 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:55.089909+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2024-09-17T09:18:56.726 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:56.090625+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2024-09-17T09:18:56.726 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:56.090715+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-17T09:18:56.726 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:56.091487+0000 mon.a (mon.0) 297 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-09-17T09:18:57.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:53.277911+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:18:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:53.278035+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:18:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:54.551870+0000 mgr.y (mgr.14144) 79 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2024-09-17T09:18:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:55.089064+0000 mon.a (mon.0) 291 : cluster [INF] osd.2 [v2:172.21.15.29:6818/4166454918,v1:172.21.15.29:6819/4166454918] boot 2024-09-17T09:18:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:55.089097+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-09-17T09:18:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:55.089427+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:18:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:55.089909+0000 mon.a (mon.0) 294 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2024-09-17T09:18:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:56.090625+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2024-09-17T09:18:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:56.090715+0000 mon.a (mon.0) 296 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-17T09:18:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:56.091487+0000 mon.a (mon.0) 297 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-09-17T09:18:57.687 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:56.552167+0000 mgr.y (mgr.14144) 80 : cluster [DBG] pgmap v65: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:18:57.687 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:18:56.616187+0000 mgr.y (mgr.14144) 81 : cephadm [INF] Detected new or changed devices on smithi029 2024-09-17T09:18:57.687 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:56.620543+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:57.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:56.623451+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:57.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:56.624617+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:57.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:56.628184+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:57.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:57.092068+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-09-17T09:18:57.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:57.092163+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-17T09:18:57.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:57.163790+0000 mgr.y (mgr.14144) 82 : audit [DBG] from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:18:57.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:57.164458+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:18:57.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:18:57.165460+0000 mgr.y (mgr.14144) 83 : cephadm [INF] Applying service osd.None on host smithi029... 2024-09-17T09:18:57.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:57.165690+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:18:57.688 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:57.166224+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:57.688 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:56.552167+0000 mgr.y (mgr.14144) 80 : cluster [DBG] pgmap v65: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:18:57.689 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:18:56.616187+0000 mgr.y (mgr.14144) 81 : cephadm [INF] Detected new or changed devices on smithi029 2024-09-17T09:18:57.689 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:56.620543+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:57.689 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:56.623451+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:57.689 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:56.624617+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:57.689 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:56.628184+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:57.689 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:57.092068+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-09-17T09:18:57.689 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:57.092163+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-17T09:18:57.689 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:57.163790+0000 mgr.y (mgr.14144) 82 : audit [DBG] from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:18:57.689 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:57.164458+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:18:57.689 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:18:57.165460+0000 mgr.y (mgr.14144) 83 : cephadm [INF] Applying service osd.None on host smithi029... 2024-09-17T09:18:57.689 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:57.165690+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:18:57.689 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:57.166224+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:18:58.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:56.552167+0000 mgr.y (mgr.14144) 80 : cluster [DBG] pgmap v65: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:18:58.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:18:56.616187+0000 mgr.y (mgr.14144) 81 : cephadm [INF] Detected new or changed devices on smithi029 2024-09-17T09:18:58.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:56.620543+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:58.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:56.623451+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:58.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:56.624617+0000 mon.a (mon.0) 300 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:18:58.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:56.628184+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:18:58.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:57.092068+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-09-17T09:18:58.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:57.092163+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-17T09:18:58.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:57.163790+0000 mgr.y (mgr.14144) 82 : audit [DBG] from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:18:58.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:57.164458+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:18:58.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:18:57.165460+0000 mgr.y (mgr.14144) 83 : cephadm [INF] Applying service osd.None on host smithi029... 2024-09-17T09:18:58.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:57.165690+0000 mon.a (mon.0) 305 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:18:58.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:57.166224+0000 mon.a (mon.0) 306 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:00.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:58.552476+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v67: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:58.624405+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-17T09:19:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:59.441402+0000 mon.a (mon.0) 308 : audit [INF] from='client.? 172.21.15.29:0/1153979722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ad1a887-9ac2-42eb-a655-e7eda6e5c26e"}]: dispatch 2024-09-17T09:19:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:59.445329+0000 mon.a (mon.0) 309 : audit [INF] from='client.? 172.21.15.29:0/1153979722' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ad1a887-9ac2-42eb-a655-e7eda6e5c26e"}]': finished 2024-09-17T09:19:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:18:59.445378+0000 mon.a (mon.0) 310 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-09-17T09:19:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:18:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:59.445496+0000 mon.a (mon.0) 311 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:00.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:58.552476+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v67: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:00.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:58.624405+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-17T09:19:00.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:59.441402+0000 mon.a (mon.0) 308 : audit [INF] from='client.? 172.21.15.29:0/1153979722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ad1a887-9ac2-42eb-a655-e7eda6e5c26e"}]: dispatch 2024-09-17T09:19:00.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:59.445329+0000 mon.a (mon.0) 309 : audit [INF] from='client.? 172.21.15.29:0/1153979722' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ad1a887-9ac2-42eb-a655-e7eda6e5c26e"}]': finished 2024-09-17T09:19:00.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:18:59.445378+0000 mon.a (mon.0) 310 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-09-17T09:19:00.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:18:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:59.445496+0000 mon.a (mon.0) 311 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:00.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:58.552476+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v67: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:00.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:58.624405+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-17T09:19:00.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:59.441402+0000 mon.a (mon.0) 308 : audit [INF] from='client.? 172.21.15.29:0/1153979722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ad1a887-9ac2-42eb-a655-e7eda6e5c26e"}]: dispatch 2024-09-17T09:19:00.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:59.445329+0000 mon.a (mon.0) 309 : audit [INF] from='client.? 172.21.15.29:0/1153979722' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ad1a887-9ac2-42eb-a655-e7eda6e5c26e"}]': finished 2024-09-17T09:19:00.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:18:59.445378+0000 mon.a (mon.0) 310 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-09-17T09:19:00.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:18:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:59.445496+0000 mon.a (mon.0) 311 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:01.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:18:59.873928+0000 mon.c (mon.1) 10 : audit [DBG] from='client.? 172.21.15.29:0/2616725973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:19:01.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:18:59.873928+0000 mon.c (mon.1) 10 : audit [DBG] from='client.? 172.21.15.29:0/2616725973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:19:01.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:18:59.873928+0000 mon.c (mon.1) 10 : audit [DBG] from='client.? 172.21.15.29:0/2616725973' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:19:02.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:00.552934+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v70: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:02.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:00.552934+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v70: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:02.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:00.552934+0000 mgr.y (mgr.14144) 85 : cluster [DBG] pgmap v70: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:04.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:02.553384+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v71: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:04.235 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:02.553384+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v71: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:04.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:02.553384+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v71: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:05.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:04.668682+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:19:05.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:04.669402+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:19:05.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:04.668682+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:19:05.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:04.669402+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:19:05.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:04.668682+0000 mon.a (mon.0) 312 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:19:05.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:04.669402+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:19:06.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:04.553672+0000 mgr.y (mgr.14144) 87 : cluster [DBG] pgmap v72: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:06.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:05.396854+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T09:19:06.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:05.397449+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:06.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:04.553672+0000 mgr.y (mgr.14144) 87 : cluster [DBG] pgmap v72: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:06.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:05.396854+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T09:19:06.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:05.397449+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:06.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:04.553672+0000 mgr.y (mgr.14144) 87 : cluster [DBG] pgmap v72: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:06.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:05.396854+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T09:19:06.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:05.397449+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:07.240 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:19:05.397957+0000 mgr.y (mgr.14144) 88 : cephadm [INF] Deploying daemon osd.3 on smithi029 2024-09-17T09:19:07.241 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:19:05.397957+0000 mgr.y (mgr.14144) 88 : cephadm [INF] Deploying daemon osd.3 on smithi029 2024-09-17T09:19:07.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:19:05.397957+0000 mgr.y (mgr.14144) 88 : cephadm [INF] Deploying daemon osd.3 on smithi029 2024-09-17T09:19:08.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:06.554105+0000 mgr.y (mgr.14144) 89 : cluster [DBG] pgmap v73: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:08.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:06.554105+0000 mgr.y (mgr.14144) 89 : cluster [DBG] pgmap v73: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:08.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:06.554105+0000 mgr.y (mgr.14144) 89 : cluster [DBG] pgmap v73: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:08.947 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 3 on host 'smithi029' 2024-09-17T09:19:09.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:08.935926+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:09.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:08.939665+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:09.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:08.944385+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:09.283 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:08.935926+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:09.283 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:08.939665+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:09.283 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:08.944385+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:09.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:08.935926+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:09.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:08.939665+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:09.284 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:08.944385+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:09.618 DEBUG:teuthology.orchestra.run.smithi029:osd.3> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.3.service 2024-09-17T09:19:09.621 INFO:tasks.cephadm:Deploying osd.4 on smithi103 with /dev/vg_nvme/lv_4... 2024-09-17T09:19:09.622 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-17T09:19:10.227 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:08.554635+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v74: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:10.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:08.554635+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v74: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:10.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:08.554635+0000 mgr.y (mgr.14144) 90 : cluster [DBG] pgmap v74: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:11.585 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-17T09:19:11.585 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-17T09:19:11.585 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-09-17T09:19:11.585 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-09-17T09:19:11.585 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129831 s, 808 MB/s 2024-09-17T09:19:11.586 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-09-17T09:19:11.749 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:10.554959+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v75: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:11.750 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:10.563429+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:11.750 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:10.554959+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v75: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:11.750 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:10.563429+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:11.928 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:10.554959+0000 mgr.y (mgr.14144) 91 : cluster [DBG] pgmap v75: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:11.928 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:10.563429+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:12.040 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:19:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.3[36462]: debug 2024-09-17T09:19:11.995+0000 7f9449bb8080 -1 osd.3 0 log_to_monitors {default=true} 2024-09-17T09:19:12.221 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_4 2024-09-17T09:19:12.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:11.999694+0000 mon.a (mon.0) 320 : audit [INF] from='osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T09:19:12.878 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:11.999694+0000 mon.a (mon.0) 320 : audit [INF] from='osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T09:19:12.878 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:11.999694+0000 mon.a (mon.0) 320 : audit [INF] from='osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T09:19:13.740 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:12.555276+0000 mgr.y (mgr.14144) 92 : cluster [DBG] pgmap v76: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:13.740 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:12.569992+0000 mon.a (mon.0) 321 : audit [INF] from='osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T09:19:13.740 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:12.570088+0000 mon.a (mon.0) 322 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-17T09:19:13.740 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:12.570308+0000 mon.a (mon.0) 323 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:13.740 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:12.570800+0000 mon.a (mon.0) 324 : audit [INF] from='osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:19:13.798 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:19:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.3[36462]: debug 2024-09-17T09:19:13.575+0000 7f9440129700 -1 osd.3 0 waiting for initial osdmap 2024-09-17T09:19:13.798 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:19:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.3[36462]: debug 2024-09-17T09:19:13.581+0000 7f943c0a4700 -1 osd.3 24 set_numa_affinity unable to identify public interface 'ens1f0' numa node: (0) Success 2024-09-17T09:19:13.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:12.555276+0000 mgr.y (mgr.14144) 92 : cluster [DBG] pgmap v76: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:13.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:12.569992+0000 mon.a (mon.0) 321 : audit [INF] from='osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T09:19:13.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:12.570088+0000 mon.a (mon.0) 322 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-17T09:19:13.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:12.570308+0000 mon.a (mon.0) 323 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:13.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:12.570800+0000 mon.a (mon.0) 324 : audit [INF] from='osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:19:13.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:12.555276+0000 mgr.y (mgr.14144) 92 : cluster [DBG] pgmap v76: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:13.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:12.569992+0000 mon.a (mon.0) 321 : audit [INF] from='osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T09:19:13.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:12.570088+0000 mon.a (mon.0) 322 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-09-17T09:19:13.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:12.570308+0000 mon.a (mon.0) 323 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:13.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:12.570800+0000 mon.a (mon.0) 324 : audit [INF] from='osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:19:14.937 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:13.572083+0000 mon.a (mon.0) 325 : audit [INF] from='osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-17T09:19:14.938 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:13.572185+0000 mon.a (mon.0) 326 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-17T09:19:14.938 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:13.572623+0000 mon.a (mon.0) 327 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:14.938 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:13.577453+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:14.938 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:19:13.678965+0000 mgr.y (mgr.14144) 93 : cephadm [INF] Detected new or changed devices on smithi029 2024-09-17T09:19:14.938 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:13.683302+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:14.938 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:13.686256+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:14.938 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:13.687453+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:14.938 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:13.690915+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:14.938 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:14.020920+0000 mgr.y (mgr.14144) 94 : audit [DBG] from='client.34181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:19:14.938 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:14.021679+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:19:14.938 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:19:14.023009+0000 mgr.y (mgr.14144) 95 : cephadm [INF] Applying service osd.None on host smithi103... 2024-09-17T09:19:14.938 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:14.023245+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:19:14.938 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:14.023864+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:13.572083+0000 mon.a (mon.0) 325 : audit [INF] from='osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-17T09:19:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:13.572185+0000 mon.a (mon.0) 326 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-17T09:19:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:13.572623+0000 mon.a (mon.0) 327 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:15.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:13.577453+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:15.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:19:13.678965+0000 mgr.y (mgr.14144) 93 : cephadm [INF] Detected new or changed devices on smithi029 2024-09-17T09:19:15.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:13.683302+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:15.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:13.686256+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:15.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:13.687453+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:15.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:13.690915+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:15.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:14.020920+0000 mgr.y (mgr.14144) 94 : audit [DBG] from='client.34181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:19:15.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:14.021679+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:19:15.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:19:14.023009+0000 mgr.y (mgr.14144) 95 : cephadm [INF] Applying service osd.None on host smithi103... 2024-09-17T09:19:15.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:14.023245+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:19:15.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:14.023864+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:15.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:13.572083+0000 mon.a (mon.0) 325 : audit [INF] from='osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-09-17T09:19:15.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:13.572185+0000 mon.a (mon.0) 326 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-09-17T09:19:15.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:13.572623+0000 mon.a (mon.0) 327 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:15.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:13.577453+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:15.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:19:13.678965+0000 mgr.y (mgr.14144) 93 : cephadm [INF] Detected new or changed devices on smithi029 2024-09-17T09:19:15.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:13.683302+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:15.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:13.686256+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:15.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:13.687453+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:15.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:13.690915+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:15.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:14.020920+0000 mgr.y (mgr.14144) 94 : audit [DBG] from='client.34181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:19:15.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:14.021679+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:19:15.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:19:14.023009+0000 mgr.y (mgr.14144) 95 : cephadm [INF] Applying service osd.None on host smithi103... 2024-09-17T09:19:15.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:14.023245+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:19:15.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:14.023864+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:16.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:12.962311+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:19:16.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:12.962434+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:19:16.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:14.555754+0000 mgr.y (mgr.14144) 96 : cluster [DBG] pgmap v79: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:16.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:14.578297+0000 mon.a (mon.0) 336 : cluster [INF] osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319] boot 2024-09-17T09:19:16.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:14.578350+0000 mon.a (mon.0) 337 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-09-17T09:19:16.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:14.578792+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:16.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:12.962311+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:19:16.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:12.962434+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:19:16.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:14.555754+0000 mgr.y (mgr.14144) 96 : cluster [DBG] pgmap v79: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:16.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:14.578297+0000 mon.a (mon.0) 336 : cluster [INF] osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319] boot 2024-09-17T09:19:16.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:14.578350+0000 mon.a (mon.0) 337 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-09-17T09:19:16.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:14.578792+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:16.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:12.962311+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:19:16.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:12.962434+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:19:16.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:14.555754+0000 mgr.y (mgr.14144) 96 : cluster [DBG] pgmap v79: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T09:19:16.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:14.578297+0000 mon.a (mon.0) 336 : cluster [INF] osd.3 [v2:172.21.15.29:6826/2996228319,v1:172.21.15.29:6827/2996228319] boot 2024-09-17T09:19:16.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:14.578350+0000 mon.a (mon.0) 337 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-09-17T09:19:16.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:14.578792+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:19:17.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:15.694638+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-09-17T09:19:17.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:16.111126+0000 mon.b (mon.2) 7 : audit [INF] from='client.? 172.21.15.103:0/327261358' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9b494dd-fa97-426e-9096-bfc1625f5826"}]: dispatch 2024-09-17T09:19:17.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:16.111586+0000 mon.a (mon.0) 340 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9b494dd-fa97-426e-9096-bfc1625f5826"}]: dispatch 2024-09-17T09:19:17.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:16.116320+0000 mon.a (mon.0) 341 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9b494dd-fa97-426e-9096-bfc1625f5826"}]': finished 2024-09-17T09:19:17.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:16.116404+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2024-09-17T09:19:17.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:16.116543+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:17.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:16.534758+0000 mon.b (mon.2) 8 : audit [DBG] from='client.? 172.21.15.103:0/3517367153' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:19:17.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:15.694638+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-09-17T09:19:17.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:16.111126+0000 mon.b (mon.2) 7 : audit [INF] from='client.? 172.21.15.103:0/327261358' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9b494dd-fa97-426e-9096-bfc1625f5826"}]: dispatch 2024-09-17T09:19:17.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:16.111586+0000 mon.a (mon.0) 340 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9b494dd-fa97-426e-9096-bfc1625f5826"}]: dispatch 2024-09-17T09:19:17.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:16.116320+0000 mon.a (mon.0) 341 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9b494dd-fa97-426e-9096-bfc1625f5826"}]': finished 2024-09-17T09:19:17.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:16.116404+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2024-09-17T09:19:17.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:16.116543+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:17.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:16.534758+0000 mon.b (mon.2) 8 : audit [DBG] from='client.? 172.21.15.103:0/3517367153' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:19:17.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:15.694638+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-09-17T09:19:17.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:16.111126+0000 mon.b (mon.2) 7 : audit [INF] from='client.? 172.21.15.103:0/327261358' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9b494dd-fa97-426e-9096-bfc1625f5826"}]: dispatch 2024-09-17T09:19:17.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:16.111586+0000 mon.a (mon.0) 340 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9b494dd-fa97-426e-9096-bfc1625f5826"}]: dispatch 2024-09-17T09:19:17.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:16.116320+0000 mon.a (mon.0) 341 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9b494dd-fa97-426e-9096-bfc1625f5826"}]': finished 2024-09-17T09:19:17.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:16.116404+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2024-09-17T09:19:17.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:16.116543+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:17.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:16.534758+0000 mon.b (mon.2) 8 : audit [DBG] from='client.? 172.21.15.103:0/3517367153' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:19:18.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:16.556119+0000 mgr.y (mgr.14144) 97 : cluster [DBG] pgmap v83: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:18.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:16.556119+0000 mgr.y (mgr.14144) 97 : cluster [DBG] pgmap v83: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:18.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:16.556119+0000 mgr.y (mgr.14144) 97 : cluster [DBG] pgmap v83: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:20.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:18.556417+0000 mgr.y (mgr.14144) 98 : cluster [DBG] pgmap v84: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:20.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:18.556417+0000 mgr.y (mgr.14144) 98 : cluster [DBG] pgmap v84: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:20.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:18.556417+0000 mgr.y (mgr.14144) 98 : cluster [DBG] pgmap v84: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:22.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:20.556847+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v85: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:22.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:21.693819+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T09:19:22.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:21.694698+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:22.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:20.556847+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v85: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:22.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:21.693819+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T09:19:22.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:21.694698+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:22.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:20.556847+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v85: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:22.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:21.693819+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T09:19:22.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:21.694698+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:23.375 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:19:21.695512+0000 mgr.y (mgr.14144) 100 : cephadm [INF] Deploying daemon osd.4 on smithi103 2024-09-17T09:19:23.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:19:21.695512+0000 mgr.y (mgr.14144) 100 : cephadm [INF] Deploying daemon osd.4 on smithi103 2024-09-17T09:19:23.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:19:21.695512+0000 mgr.y (mgr.14144) 100 : cephadm [INF] Deploying daemon osd.4 on smithi103 2024-09-17T09:19:24.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:22.557117+0000 mgr.y (mgr.14144) 101 : cluster [DBG] pgmap v86: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:24.450 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:22.557117+0000 mgr.y (mgr.14144) 101 : cluster [DBG] pgmap v86: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:24.458 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:22.557117+0000 mgr.y (mgr.14144) 101 : cluster [DBG] pgmap v86: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:24.855 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 4 on host 'smithi103' 2024-09-17T09:19:25.422 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:24.841213+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:25.422 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:24.845475+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:25.423 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:24.849982+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:25.445 DEBUG:teuthology.orchestra.run.smithi103:osd.4> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.4.service 2024-09-17T09:19:25.447 INFO:tasks.cephadm:Deploying osd.5 on smithi103 with /dev/vg_nvme/lv_3... 2024-09-17T09:19:25.448 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-17T09:19:25.469 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:24.841213+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:25.469 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:24.845475+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:25.469 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:24.849982+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:25.469 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:24.841213+0000 mon.a (mon.0) 346 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:25.469 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:24.845475+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:25.470 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:24.849982+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:26.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:24.557478+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v87: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:26.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:24.557478+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v87: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:26.453 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:24.557478+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v87: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:28.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:26.557779+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:28.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:27.076755+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:28.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:27.956136+0000 mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.103:6800/1076748350,v1:172.21.15.103:6801/1076748350]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:19:28.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:27.956564+0000 mon.a (mon.0) 350 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:19:28.256 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:19:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.4[26604]: debug 2024-09-17T09:19:27.951+0000 7f089a6dc080 -1 osd.4 0 log_to_monitors {default=true} 2024-09-17T09:19:28.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:26.557779+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:28.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:27.076755+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:28.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:27.956136+0000 mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.103:6800/1076748350,v1:172.21.15.103:6801/1076748350]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:19:28.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:27.956564+0000 mon.a (mon.0) 350 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:19:28.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:26.557779+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v88: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:28.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:27.076755+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:28.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:27.956136+0000 mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.103:6800/1076748350,v1:172.21.15.103:6801/1076748350]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:19:28.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:27.956564+0000 mon.a (mon.0) 350 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:19:28.715 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-17T09:19:28.715 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-17T09:19:28.715 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-09-17T09:19:28.715 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-09-17T09:19:28.716 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0139915 s, 749 MB/s 2024-09-17T09:19:28.716 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-09-17T09:19:29.505 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:19:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.4[26604]: debug 2024-09-17T09:19:29.144+0000 7f0890c4d700 -1 osd.4 0 waiting for initial osdmap 2024-09-17T09:19:29.506 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:19:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.4[26604]: debug 2024-09-17T09:19:29.149+0000 7f088b3c5700 -1 osd.4 29 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-09-17T09:19:29.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:28.138669+0000 mon.a (mon.0) 351 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T09:19:29.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:28.138781+0000 mon.a (mon.0) 352 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-09-17T09:19:29.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:28.138965+0000 mon.a (mon.0) 353 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:29.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:28.140637+0000 mon.b (mon.2) 10 : audit [INF] from='osd.4 [v2:172.21.15.103:6800/1076748350,v1:172.21.15.103:6801/1076748350]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:19:29.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:28.140954+0000 mon.a (mon.0) 354 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:19:29.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:28.138669+0000 mon.a (mon.0) 351 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T09:19:29.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:28.138781+0000 mon.a (mon.0) 352 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-09-17T09:19:29.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:28.138965+0000 mon.a (mon.0) 353 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:29.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:28.140637+0000 mon.b (mon.2) 10 : audit [INF] from='osd.4 [v2:172.21.15.103:6800/1076748350,v1:172.21.15.103:6801/1076748350]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:19:29.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:28.140954+0000 mon.a (mon.0) 354 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:19:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:28.138669+0000 mon.a (mon.0) 351 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T09:19:29.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:28.138781+0000 mon.a (mon.0) 352 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-09-17T09:19:29.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:28.138965+0000 mon.a (mon.0) 353 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:29.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:28.140637+0000 mon.b (mon.2) 10 : audit [INF] from='osd.4 [v2:172.21.15.103:6800/1076748350,v1:172.21.15.103:6801/1076748350]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:19:29.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:28.140954+0000 mon.a (mon.0) 354 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:19:29.606 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_3 2024-09-17T09:19:30.452 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:28.558049+0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:30.452 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:29.141092+0000 mon.a (mon.0) 355 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-09-17T09:19:30.452 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:29.141216+0000 mon.a (mon.0) 356 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-09-17T09:19:30.452 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:29.141794+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:30.452 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:29.145799+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:30.452 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:30.142615+0000 mon.a (mon.0) 359 : cluster [INF] osd.4 [v2:172.21.15.103:6800/1076748350,v1:172.21.15.103:6801/1076748350] boot 2024-09-17T09:19:30.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:28.558049+0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:30.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:29.141092+0000 mon.a (mon.0) 355 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-09-17T09:19:30.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:29.141216+0000 mon.a (mon.0) 356 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-09-17T09:19:30.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:29.141794+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:30.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:29.145799+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:30.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:30.142615+0000 mon.a (mon.0) 359 : cluster [INF] osd.4 [v2:172.21.15.103:6800/1076748350,v1:172.21.15.103:6801/1076748350] boot 2024-09-17T09:19:30.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:28.558049+0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v90: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-09-17T09:19:30.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:29.141092+0000 mon.a (mon.0) 355 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-09-17T09:19:30.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:29.141216+0000 mon.a (mon.0) 356 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-09-17T09:19:30.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:29.141794+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:30.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:29.145799+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:30.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:30.142615+0000 mon.a (mon.0) 359 : cluster [INF] osd.4 [v2:172.21.15.103:6800/1076748350,v1:172.21.15.103:6801/1076748350] boot 2024-09-17T09:19:31.404 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:28.980978+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:19:31.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:28.981163+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:19:31.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:30.142709+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2024-09-17T09:19:31.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:30.142916+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:31.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:19:30.381548+0000 mgr.y (mgr.14144) 105 : cephadm [INF] Detected new or changed devices on smithi103 2024-09-17T09:19:31.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:30.386374+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:31.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:30.602587+0000 mon.a (mon.0) 363 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:31.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:30.604354+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:31.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:30.609259+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:31.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:31.143367+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-09-17T09:19:31.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:28.980978+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:19:31.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:28.981163+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:19:31.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:30.142709+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2024-09-17T09:19:31.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:30.142916+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:19:30.381548+0000 mgr.y (mgr.14144) 105 : cephadm [INF] Detected new or changed devices on smithi103 2024-09-17T09:19:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:30.386374+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:30.602587+0000 mon.a (mon.0) 363 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:30.604354+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:30.609259+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:31.143367+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-09-17T09:19:31.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:28.980978+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:19:31.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:28.981163+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:19:31.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:30.142709+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2024-09-17T09:19:31.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:30.142916+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:19:31.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:19:30.381548+0000 mgr.y (mgr.14144) 105 : cephadm [INF] Detected new or changed devices on smithi103 2024-09-17T09:19:31.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:30.386374+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:31.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:30.602587+0000 mon.a (mon.0) 363 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:31.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:30.604354+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:31.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:30.609259+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:31.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:31.143367+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-09-17T09:19:32.456 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:30.558417+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:32.456 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:31.902569+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:19:32.456 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:31.904234+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:19:32.456 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:31.905000+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:32.456 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:32.144617+0000 mon.a (mon.0) 370 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-09-17T09:19:32.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:30.558417+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:32.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:31.902569+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:19:32.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:31.904234+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:19:32.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:31.905000+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:32.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:32.144617+0000 mon.a (mon.0) 370 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-09-17T09:19:32.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:30.558417+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:32.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:31.902569+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:19:32.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:31.904234+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:19:32.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:31.905000+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:32.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:32.144617+0000 mon.a (mon.0) 370 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-09-17T09:19:33.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:31.901688+0000 mgr.y (mgr.14144) 107 : audit [DBG] from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:19:33.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:19:31.903986+0000 mgr.y (mgr.14144) 108 : cephadm [INF] Applying service osd.None on host smithi103... 2024-09-17T09:19:33.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:31.901688+0000 mgr.y (mgr.14144) 107 : audit [DBG] from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:19:33.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:19:31.903986+0000 mgr.y (mgr.14144) 108 : cephadm [INF] Applying service osd.None on host smithi103... 2024-09-17T09:19:33.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:31.901688+0000 mgr.y (mgr.14144) 107 : audit [DBG] from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:19:33.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:19:31.903986+0000 mgr.y (mgr.14144) 108 : cephadm [INF] Applying service osd.None on host smithi103... 2024-09-17T09:19:34.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:32.558692+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v96: 1 pgs: 1 remapped+peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:34.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:34.113734+0000 mon.b (mon.2) 11 : audit [INF] from='client.? 172.21.15.103:0/329353115' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78565831-70c9-4dc9-85e2-06accbc9a1bb"}]: dispatch 2024-09-17T09:19:34.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:34.114277+0000 mon.a (mon.0) 371 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78565831-70c9-4dc9-85e2-06accbc9a1bb"}]: dispatch 2024-09-17T09:19:34.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:34.119831+0000 mon.a (mon.0) 372 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78565831-70c9-4dc9-85e2-06accbc9a1bb"}]': finished 2024-09-17T09:19:34.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:34.119921+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2024-09-17T09:19:34.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:34.120067+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:34.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:32.558692+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v96: 1 pgs: 1 remapped+peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:34.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:34.113734+0000 mon.b (mon.2) 11 : audit [INF] from='client.? 172.21.15.103:0/329353115' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78565831-70c9-4dc9-85e2-06accbc9a1bb"}]: dispatch 2024-09-17T09:19:34.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:34.114277+0000 mon.a (mon.0) 371 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78565831-70c9-4dc9-85e2-06accbc9a1bb"}]: dispatch 2024-09-17T09:19:34.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:34.119831+0000 mon.a (mon.0) 372 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78565831-70c9-4dc9-85e2-06accbc9a1bb"}]': finished 2024-09-17T09:19:34.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:34.119921+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2024-09-17T09:19:34.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:34.120067+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:34.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:32.558692+0000 mgr.y (mgr.14144) 109 : cluster [DBG] pgmap v96: 1 pgs: 1 remapped+peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:34.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:34.113734+0000 mon.b (mon.2) 11 : audit [INF] from='client.? 172.21.15.103:0/329353115' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78565831-70c9-4dc9-85e2-06accbc9a1bb"}]: dispatch 2024-09-17T09:19:34.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:34.114277+0000 mon.a (mon.0) 371 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78565831-70c9-4dc9-85e2-06accbc9a1bb"}]: dispatch 2024-09-17T09:19:34.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:34.119831+0000 mon.a (mon.0) 372 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78565831-70c9-4dc9-85e2-06accbc9a1bb"}]': finished 2024-09-17T09:19:34.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:34.119921+0000 mon.a (mon.0) 373 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2024-09-17T09:19:34.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:34.120067+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:35.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:34.527624+0000 mon.b (mon.2) 12 : audit [DBG] from='client.? 172.21.15.103:0/1054821730' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:19:35.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:34.559006+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v98: 1 pgs: 1 remapped+peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:35.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:34.527624+0000 mon.b (mon.2) 12 : audit [DBG] from='client.? 172.21.15.103:0/1054821730' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:19:35.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:34.559006+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v98: 1 pgs: 1 remapped+peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:35.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:34.527624+0000 mon.b (mon.2) 12 : audit [DBG] from='client.? 172.21.15.103:0/1054821730' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:19:35.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:34.559006+0000 mgr.y (mgr.14144) 110 : cluster [DBG] pgmap v98: 1 pgs: 1 remapped+peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:37.952 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:36.559520+0000 mgr.y (mgr.14144) 111 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:38.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:36.559520+0000 mgr.y (mgr.14144) 111 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:38.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:36.559520+0000 mgr.y (mgr.14144) 111 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:40.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:38.559890+0000 mgr.y (mgr.14144) 112 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:40.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:39.777306+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T09:19:40.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:39.778250+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:40.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:38.559890+0000 mgr.y (mgr.14144) 112 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:40.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:39.777306+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T09:19:40.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:39.778250+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:40.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:38.559890+0000 mgr.y (mgr.14144) 112 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:40.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:39.777306+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T09:19:40.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:39.778250+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:41.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:19:39.778943+0000 mgr.y (mgr.14144) 113 : cephadm [INF] Deploying daemon osd.5 on smithi103 2024-09-17T09:19:41.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:19:39.778943+0000 mgr.y (mgr.14144) 113 : cephadm [INF] Deploying daemon osd.5 on smithi103 2024-09-17T09:19:41.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:19:39.778943+0000 mgr.y (mgr.14144) 113 : cephadm [INF] Deploying daemon osd.5 on smithi103 2024-09-17T09:19:42.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:40.560185+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:42.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:40.560185+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:42.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:40.560185+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:43.303 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 5 on host 'smithi103' 2024-09-17T09:19:43.912 DEBUG:teuthology.orchestra.run.smithi103:osd.5> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.5.service 2024-09-17T09:19:43.915 INFO:tasks.cephadm:Deploying osd.6 on smithi103 with /dev/vg_nvme/lv_2... 2024-09-17T09:19:43.915 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-17T09:19:44.239 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:42.560538+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:44.239 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:43.290975+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:44.239 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:43.294933+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:44.239 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:43.300112+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:44.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:42.560538+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:44.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:43.290975+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:44.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:43.294933+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:44.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:43.300112+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:44.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:42.560538+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:44.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:43.290975+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:44.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:43.294933+0000 mon.a (mon.0) 378 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:44.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:43.300112+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:46.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:44.560916+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:46.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:45.896176+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:46.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:44.560916+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:46.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:45.896176+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:46.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:44.560916+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:46.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:45.896176+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:46.418 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-17T09:19:46.418 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-17T09:19:46.418 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-09-17T09:19:46.418 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-09-17T09:19:46.418 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0143082 s, 733 MB/s 2024-09-17T09:19:46.418 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-09-17T09:19:46.755 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:19:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.5[28976]: debug 2024-09-17T09:19:46.345+0000 7fe02e92f080 -1 osd.5 0 log_to_monitors {default=true} 2024-09-17T09:19:47.041 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:46.350115+0000 mon.b (mon.2) 13 : audit [INF] from='osd.5 [v2:172.21.15.103:6808/902800869,v1:172.21.15.103:6809/902800869]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:19:47.041 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:46.350478+0000 mon.a (mon.0) 381 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:19:47.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:46.350115+0000 mon.b (mon.2) 13 : audit [INF] from='osd.5 [v2:172.21.15.103:6808/902800869,v1:172.21.15.103:6809/902800869]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:19:47.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:46.350478+0000 mon.a (mon.0) 381 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:19:47.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:46.350115+0000 mon.b (mon.2) 13 : audit [INF] from='osd.5 [v2:172.21.15.103:6808/902800869,v1:172.21.15.103:6809/902800869]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:19:47.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:46.350478+0000 mon.a (mon.0) 381 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:19:48.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_2 2024-09-17T09:19:48.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:46.561324+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:48.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:47.006193+0000 mon.a (mon.0) 382 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T09:19:48.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:47.006293+0000 mon.a (mon.0) 383 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-09-17T09:19:48.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:47.006597+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:48.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:47.006872+0000 mon.b (mon.2) 14 : audit [INF] from='osd.5 [v2:172.21.15.103:6808/902800869,v1:172.21.15.103:6809/902800869]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:19:48.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:47.007181+0000 mon.a (mon.0) 385 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:19:48.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:46.561324+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:48.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:47.006193+0000 mon.a (mon.0) 382 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T09:19:48.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:47.006293+0000 mon.a (mon.0) 383 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-09-17T09:19:48.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:47.006597+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:48.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:47.006872+0000 mon.b (mon.2) 14 : audit [INF] from='osd.5 [v2:172.21.15.103:6808/902800869,v1:172.21.15.103:6809/902800869]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:19:48.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:47.007181+0000 mon.a (mon.0) 385 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:19:48.342 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:19:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.5[28976]: debug 2024-09-17T09:19:48.009+0000 7fe024ea0700 -1 osd.5 0 waiting for initial osdmap 2024-09-17T09:19:48.342 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:19:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.5[28976]: debug 2024-09-17T09:19:48.014+0000 7fe02161c700 -1 osd.5 35 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-09-17T09:19:48.343 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:46.561324+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:48.343 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:47.006193+0000 mon.a (mon.0) 382 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T09:19:48.343 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:47.006293+0000 mon.a (mon.0) 383 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-09-17T09:19:48.343 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:47.006597+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:48.343 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:47.006872+0000 mon.b (mon.2) 14 : audit [INF] from='osd.5 [v2:172.21.15.103:6808/902800869,v1:172.21.15.103:6809/902800869]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:19:48.343 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:47.007181+0000 mon.a (mon.0) 385 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:19:49.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:48.007626+0000 mon.a (mon.0) 386 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-09-17T09:19:49.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:48.007798+0000 mon.a (mon.0) 387 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-09-17T09:19:49.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:48.008302+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:49.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:48.012958+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:49.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:48.007626+0000 mon.a (mon.0) 386 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-09-17T09:19:49.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:48.007798+0000 mon.a (mon.0) 387 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-09-17T09:19:49.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:48.008302+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:49.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:48.012958+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:49.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:48.007626+0000 mon.a (mon.0) 386 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-09-17T09:19:49.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:48.007798+0000 mon.a (mon.0) 387 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-09-17T09:19:49.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:48.008302+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:49.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:48.012958+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:50.181 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:47.362047+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:19:50.181 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:47.362182+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:19:50.181 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:48.561805+0000 mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:50.181 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:49.013178+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:50.181 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:49.017217+0000 mon.a (mon.0) 391 : cluster [INF] osd.5 [v2:172.21.15.103:6808/902800869,v1:172.21.15.103:6809/902800869] boot 2024-09-17T09:19:50.181 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:49.017285+0000 mon.a (mon.0) 392 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2024-09-17T09:19:50.181 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:49.017761+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:50.181 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:49.751718+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:50.181 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:49.755511+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:50.182 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:49.757501+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:50.182 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:49.762190+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:47.362047+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:19:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:47.362182+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:19:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:48.561805+0000 mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:49.013178+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:49.017217+0000 mon.a (mon.0) 391 : cluster [INF] osd.5 [v2:172.21.15.103:6808/902800869,v1:172.21.15.103:6809/902800869] boot 2024-09-17T09:19:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:49.017285+0000 mon.a (mon.0) 392 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2024-09-17T09:19:50.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:49.017761+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:50.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:49.751718+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:50.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:49.755511+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:50.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:49.757501+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:50.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:49.762190+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:50.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:47.362047+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:19:50.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:47.362182+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:19:50.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:48.561805+0000 mgr.y (mgr.14144) 118 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-09-17T09:19:50.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:49.013178+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:50.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:49.017217+0000 mon.a (mon.0) 391 : cluster [INF] osd.5 [v2:172.21.15.103:6808/902800869,v1:172.21.15.103:6809/902800869] boot 2024-09-17T09:19:50.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:49.017285+0000 mon.a (mon.0) 392 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2024-09-17T09:19:50.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:49.017761+0000 mon.a (mon.0) 393 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:19:50.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:49.751718+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:50.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:49.755511+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:50.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:49.757501+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:19:50.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:49.762190+0000 mon.a (mon.0) 397 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:19:51.122 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:19:49.747053+0000 mgr.y (mgr.14144) 119 : cephadm [INF] Detected new or changed devices on smithi103 2024-09-17T09:19:51.122 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:50.018191+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-09-17T09:19:51.122 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:50.588578+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:19:51.123 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:50.590206+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:19:51.123 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:50.591017+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:51.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:19:49.747053+0000 mgr.y (mgr.14144) 119 : cephadm [INF] Detected new or changed devices on smithi103 2024-09-17T09:19:51.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:50.018191+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-09-17T09:19:51.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:50.588578+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:19:51.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:50.590206+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:19:51.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:50.591017+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:51.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:19:49.747053+0000 mgr.y (mgr.14144) 119 : cephadm [INF] Detected new or changed devices on smithi103 2024-09-17T09:19:51.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:50.018191+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-09-17T09:19:51.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:50.588578+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:19:51.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:50.590206+0000 mon.a (mon.0) 400 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:19:51.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:50.591017+0000 mon.a (mon.0) 401 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:52.095 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:50.562252+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:19:52.095 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:50.587734+0000 mgr.y (mgr.14144) 121 : audit [DBG] from='client.34235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:19:52.095 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:19:50.589968+0000 mgr.y (mgr.14144) 122 : cephadm [INF] Applying service osd.None on host smithi103... 2024-09-17T09:19:52.095 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:51.020081+0000 mon.a (mon.0) 402 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-09-17T09:19:52.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:50.562252+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:19:52.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:50.587734+0000 mgr.y (mgr.14144) 121 : audit [DBG] from='client.34235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:19:52.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:19:50.589968+0000 mgr.y (mgr.14144) 122 : cephadm [INF] Applying service osd.None on host smithi103... 2024-09-17T09:19:52.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:51.020081+0000 mon.a (mon.0) 402 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-09-17T09:19:52.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:50.562252+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:19:52.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:50.587734+0000 mgr.y (mgr.14144) 121 : audit [DBG] from='client.34235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:19:52.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:19:50.589968+0000 mgr.y (mgr.14144) 122 : cephadm [INF] Applying service osd.None on host smithi103... 2024-09-17T09:19:52.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:51.020081+0000 mon.a (mon.0) 402 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-09-17T09:19:53.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:52.637756+0000 mon.b (mon.2) 15 : audit [INF] from='client.? 172.21.15.103:0/3002633365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c84571-a2b3-48c3-ad16-71b144e8e332"}]: dispatch 2024-09-17T09:19:53.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:52.638175+0000 mon.a (mon.0) 403 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c84571-a2b3-48c3-ad16-71b144e8e332"}]: dispatch 2024-09-17T09:19:53.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:52.642964+0000 mon.a (mon.0) 404 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3c84571-a2b3-48c3-ad16-71b144e8e332"}]': finished 2024-09-17T09:19:53.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:52.643037+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e39: 7 total, 6 up, 7 in 2024-09-17T09:19:53.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:52.643176+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:19:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:52.637756+0000 mon.b (mon.2) 15 : audit [INF] from='client.? 172.21.15.103:0/3002633365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c84571-a2b3-48c3-ad16-71b144e8e332"}]: dispatch 2024-09-17T09:19:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:52.638175+0000 mon.a (mon.0) 403 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c84571-a2b3-48c3-ad16-71b144e8e332"}]: dispatch 2024-09-17T09:19:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:52.642964+0000 mon.a (mon.0) 404 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3c84571-a2b3-48c3-ad16-71b144e8e332"}]': finished 2024-09-17T09:19:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:52.643037+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e39: 7 total, 6 up, 7 in 2024-09-17T09:19:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:52.643176+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:19:53.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:52.637756+0000 mon.b (mon.2) 15 : audit [INF] from='client.? 172.21.15.103:0/3002633365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c84571-a2b3-48c3-ad16-71b144e8e332"}]: dispatch 2024-09-17T09:19:53.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:52.638175+0000 mon.a (mon.0) 403 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c84571-a2b3-48c3-ad16-71b144e8e332"}]: dispatch 2024-09-17T09:19:53.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:52.642964+0000 mon.a (mon.0) 404 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3c84571-a2b3-48c3-ad16-71b144e8e332"}]': finished 2024-09-17T09:19:53.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:52.643037+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e39: 7 total, 6 up, 7 in 2024-09-17T09:19:53.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:52.643176+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:19:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:52.562619+0000 mgr.y (mgr.14144) 123 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:19:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:53.055595+0000 mon.b (mon.2) 16 : audit [DBG] from='client.? 172.21.15.103:0/1587527901' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:19:54.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:52.562619+0000 mgr.y (mgr.14144) 123 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:19:54.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:53.055595+0000 mon.b (mon.2) 16 : audit [DBG] from='client.? 172.21.15.103:0/1587527901' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:19:54.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:52.562619+0000 mgr.y (mgr.14144) 123 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:19:54.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:53.055595+0000 mon.b (mon.2) 16 : audit [DBG] from='client.? 172.21.15.103:0/1587527901' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:19:56.141 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:54.563016+0000 mgr.y (mgr.14144) 124 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:19:56.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:54.563016+0000 mgr.y (mgr.14144) 124 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:19:56.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:54.563016+0000 mgr.y (mgr.14144) 124 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:19:58.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:56.563415+0000 mgr.y (mgr.14144) 125 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:19:58.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:56.563415+0000 mgr.y (mgr.14144) 125 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:19:58.353 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:56.563415+0000 mgr.y (mgr.14144) 125 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:19:59.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:58.376856+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T09:19:59.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:19:58.377765+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:59.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:19:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:19:58.378485+0000 mgr.y (mgr.14144) 126 : cephadm [INF] Deploying daemon osd.6 on smithi103 2024-09-17T09:19:59.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:58.376856+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T09:19:59.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:19:58.377765+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:59.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:19:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:19:58.378485+0000 mgr.y (mgr.14144) 126 : cephadm [INF] Deploying daemon osd.6 on smithi103 2024-09-17T09:19:59.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:58.376856+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T09:19:59.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:19:58.377765+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:19:59.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:19:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:19:58.378485+0000 mgr.y (mgr.14144) 126 : cephadm [INF] Deploying daemon osd.6 on smithi103 2024-09-17T09:20:00.232 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:19:58.563899+0000 mgr.y (mgr.14144) 127 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:00.232 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:00.000131+0000 mon.a (mon.0) 409 : cluster [INF] overall HEALTH_OK 2024-09-17T09:20:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:19:58.563899+0000 mgr.y (mgr.14144) 127 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:00.000131+0000 mon.a (mon.0) 409 : cluster [INF] overall HEALTH_OK 2024-09-17T09:20:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:19:58.563899+0000 mgr.y (mgr.14144) 127 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:00.000131+0000 mon.a (mon.0) 409 : cluster [INF] overall HEALTH_OK 2024-09-17T09:20:01.731 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 6 on host 'smithi103' 2024-09-17T09:20:02.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:00.564247+0000 mgr.y (mgr.14144) 128 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:02.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:01.719329+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:02.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:01.724005+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:02.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:01.728960+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:02.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:00.564247+0000 mgr.y (mgr.14144) 128 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:02.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:01.719329+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:02.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:01.724005+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:02.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:01.728960+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:02.316 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:00.564247+0000 mgr.y (mgr.14144) 128 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:02.316 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:01.719329+0000 mon.a (mon.0) 410 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:02.316 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:01.724005+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:02.317 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:01.728960+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:02.345 DEBUG:teuthology.orchestra.run.smithi103:osd.6> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.6.service 2024-09-17T09:20:02.347 INFO:tasks.cephadm:Deploying osd.7 on smithi103 with /dev/vg_nvme/lv_1... 2024-09-17T09:20:02.348 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-17T09:20:04.346 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:02.564591+0000 mgr.y (mgr.14144) 129 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:04.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:02.564591+0000 mgr.y (mgr.14144) 129 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:04.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:02.564591+0000 mgr.y (mgr.14144) 129 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:04.769 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-17T09:20:04.770 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-17T09:20:04.770 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-09-17T09:20:04.770 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-09-17T09:20:04.770 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130834 s, 801 MB/s 2024-09-17T09:20:04.770 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-09-17T09:20:05.005 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:20:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.6[31353]: debug 2024-09-17T09:20:04.862+0000 7fe2a86af080 -1 osd.6 0 log_to_monitors {default=true} 2024-09-17T09:20:05.557 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_1 2024-09-17T09:20:05.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:04.497352+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:04.565032+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:04.669787+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:20:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:04.670898+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:20:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:04.865913+0000 mon.b (mon.2) 17 : audit [INF] from='osd.6 [v2:172.21.15.103:6816/3547403473,v1:172.21.15.103:6817/3547403473]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:20:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:04.866218+0000 mon.a (mon.0) 416 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:20:05.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:04.497352+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:05.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:04.565032+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:05.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:04.669787+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:20:05.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:04.670898+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:20:05.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:04.865913+0000 mon.b (mon.2) 17 : audit [INF] from='osd.6 [v2:172.21.15.103:6816/3547403473,v1:172.21.15.103:6817/3547403473]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:20:05.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:04.866218+0000 mon.a (mon.0) 416 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:20:05.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:04.497352+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:05.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:04.565032+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:05.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:04.669787+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:20:05.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:04.670898+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:20:05.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:04.865913+0000 mon.b (mon.2) 17 : audit [INF] from='osd.6 [v2:172.21.15.103:6816/3547403473,v1:172.21.15.103:6817/3547403473]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:20:05.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:04.866218+0000 mon.a (mon.0) 416 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:20:06.755 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:20:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.6[31353]: debug 2024-09-17T09:20:06.510+0000 7fe29ec20700 -1 osd.6 0 waiting for initial osdmap 2024-09-17T09:20:06.755 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:20:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.6[31353]: debug 2024-09-17T09:20:06.517+0000 7fe298b97700 -1 osd.6 41 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-09-17T09:20:06.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:05.504173+0000 mon.a (mon.0) 417 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T09:20:06.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:05.504353+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2024-09-17T09:20:06.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:05.504696+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:06.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:05.504856+0000 mon.b (mon.2) 18 : audit [INF] from='osd.6 [v2:172.21.15.103:6816/3547403473,v1:172.21.15.103:6817/3547403473]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:20:06.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:05.505477+0000 mon.a (mon.0) 420 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:20:06.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:05.504173+0000 mon.a (mon.0) 417 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T09:20:06.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:05.504353+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2024-09-17T09:20:06.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:05.504696+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:06.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:05.504856+0000 mon.b (mon.2) 18 : audit [INF] from='osd.6 [v2:172.21.15.103:6816/3547403473,v1:172.21.15.103:6817/3547403473]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:20:06.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:05.505477+0000 mon.a (mon.0) 420 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:20:06.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:05.504173+0000 mon.a (mon.0) 417 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T09:20:06.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:05.504353+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2024-09-17T09:20:06.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:05.504696+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:06.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:05.504856+0000 mon.b (mon.2) 18 : audit [INF] from='osd.6 [v2:172.21.15.103:6816/3547403473,v1:172.21.15.103:6817/3547403473]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:20:06.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:05.505477+0000 mon.a (mon.0) 420 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:20:07.725 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:06.505796+0000 mon.a (mon.0) 421 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-09-17T09:20:07.725 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:06.505901+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-09-17T09:20:07.725 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:06.506319+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:07.725 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:06.511113+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:07.725 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:06.565360+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:07.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:06.505796+0000 mon.a (mon.0) 421 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-09-17T09:20:07.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:06.505901+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-09-17T09:20:07.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:06.506319+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:07.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:06.511113+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:07.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:06.565360+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:07.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:06.505796+0000 mon.a (mon.0) 421 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-09-17T09:20:07.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:06.505901+0000 mon.a (mon.0) 422 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-09-17T09:20:07.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:06.506319+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:07.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:06.511113+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:07.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:06.565360+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2024-09-17T09:20:08.795 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:05.885669+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:20:08.795 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:05.885756+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:20:08.795 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:07.511629+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:08.795 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:07.516224+0000 mon.a (mon.0) 426 : cluster [INF] osd.6 [v2:172.21.15.103:6816/3547403473,v1:172.21.15.103:6817/3547403473] boot 2024-09-17T09:20:08.796 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:07.516295+0000 mon.a (mon.0) 427 : cluster [DBG] osdmap e42: 7 total, 7 up, 7 in 2024-09-17T09:20:08.796 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:07.516781+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:08.796 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:07.993461+0000 mon.a (mon.0) 429 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2024-09-17T09:20:08.796 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:20:08.378672+0000 mgr.y (mgr.14144) 132 : cephadm [INF] Detected new or changed devices on smithi103 2024-09-17T09:20:08.796 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:08.383756+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:08.796 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:08.387835+0000 mon.a (mon.0) 431 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:08.796 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:08.389589+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:08.796 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:08.394268+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:08.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:05.885669+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:20:08.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:05.885756+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:20:08.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:07.511629+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:08.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:07.516224+0000 mon.a (mon.0) 426 : cluster [INF] osd.6 [v2:172.21.15.103:6816/3547403473,v1:172.21.15.103:6817/3547403473] boot 2024-09-17T09:20:08.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:07.516295+0000 mon.a (mon.0) 427 : cluster [DBG] osdmap e42: 7 total, 7 up, 7 in 2024-09-17T09:20:08.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:07.516781+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:08.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:07.993461+0000 mon.a (mon.0) 429 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2024-09-17T09:20:08.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:20:08.378672+0000 mgr.y (mgr.14144) 132 : cephadm [INF] Detected new or changed devices on smithi103 2024-09-17T09:20:08.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:08.383756+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:08.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:08.387835+0000 mon.a (mon.0) 431 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:08.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:08.389589+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:08.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:08.394268+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:08.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:05.885669+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:20:08.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:05.885756+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:20:08.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:07.511629+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:08.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:07.516224+0000 mon.a (mon.0) 426 : cluster [INF] osd.6 [v2:172.21.15.103:6816/3547403473,v1:172.21.15.103:6817/3547403473] boot 2024-09-17T09:20:08.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:07.516295+0000 mon.a (mon.0) 427 : cluster [DBG] osdmap e42: 7 total, 7 up, 7 in 2024-09-17T09:20:08.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:07.516781+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:20:08.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:07.993461+0000 mon.a (mon.0) 429 : cluster [DBG] osdmap e43: 7 total, 7 up, 7 in 2024-09-17T09:20:08.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:20:08.378672+0000 mgr.y (mgr.14144) 132 : cephadm [INF] Detected new or changed devices on smithi103 2024-09-17T09:20:08.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:08.383756+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:08.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:08.387835+0000 mon.a (mon.0) 431 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:08.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:08.389589+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:08.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:08.394268+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:10.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:08.565792+0000 mgr.y (mgr.14144) 133 : cluster [DBG] pgmap v125: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:10.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:08.998473+0000 mon.a (mon.0) 434 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2024-09-17T09:20:10.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:09.203977+0000 mgr.y (mgr.14144) 134 : audit [DBG] from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:10.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:09.204864+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:20:10.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:20:09.206236+0000 mgr.y (mgr.14144) 135 : cephadm [INF] Applying service osd.None on host smithi103... 2024-09-17T09:20:10.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:09.206497+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:20:10.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:09.207251+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:20:10.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:08.565792+0000 mgr.y (mgr.14144) 133 : cluster [DBG] pgmap v125: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:10.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:08.998473+0000 mon.a (mon.0) 434 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2024-09-17T09:20:10.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:09.203977+0000 mgr.y (mgr.14144) 134 : audit [DBG] from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:10.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:09.204864+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:20:10.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:20:09.206236+0000 mgr.y (mgr.14144) 135 : cephadm [INF] Applying service osd.None on host smithi103... 2024-09-17T09:20:10.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:09.206497+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:20:10.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:09.207251+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:20:10.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:08.565792+0000 mgr.y (mgr.14144) 133 : cluster [DBG] pgmap v125: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:10.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:08.998473+0000 mon.a (mon.0) 434 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2024-09-17T09:20:10.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:09.203977+0000 mgr.y (mgr.14144) 134 : audit [DBG] from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:10.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:09.204864+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T09:20:10.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:20:09.206236+0000 mgr.y (mgr.14144) 135 : cephadm [INF] Applying service osd.None on host smithi103... 2024-09-17T09:20:10.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:09.206497+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T09:20:10.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:09.207251+0000 mon.a (mon.0) 437 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:20:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:10.566133+0000 mgr.y (mgr.14144) 136 : cluster [DBG] pgmap v127: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:11.308311+0000 mon.b (mon.2) 19 : audit [INF] from='client.? 172.21.15.103:0/3520111888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5313bab3-7c30-497b-9524-a3c63851c73a"}]: dispatch 2024-09-17T09:20:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:11.308756+0000 mon.a (mon.0) 438 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5313bab3-7c30-497b-9524-a3c63851c73a"}]: dispatch 2024-09-17T09:20:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:11.313517+0000 mon.a (mon.0) 439 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5313bab3-7c30-497b-9524-a3c63851c73a"}]': finished 2024-09-17T09:20:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:11.313617+0000 mon.a (mon.0) 440 : cluster [DBG] osdmap e45: 8 total, 7 up, 8 in 2024-09-17T09:20:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:11.313768+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:11.728475+0000 mon.b (mon.2) 20 : audit [DBG] from='client.? 172.21.15.103:0/807844062' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:20:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:10.566133+0000 mgr.y (mgr.14144) 136 : cluster [DBG] pgmap v127: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:11.308311+0000 mon.b (mon.2) 19 : audit [INF] from='client.? 172.21.15.103:0/3520111888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5313bab3-7c30-497b-9524-a3c63851c73a"}]: dispatch 2024-09-17T09:20:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:11.308756+0000 mon.a (mon.0) 438 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5313bab3-7c30-497b-9524-a3c63851c73a"}]: dispatch 2024-09-17T09:20:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:11.313517+0000 mon.a (mon.0) 439 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5313bab3-7c30-497b-9524-a3c63851c73a"}]': finished 2024-09-17T09:20:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:11.313617+0000 mon.a (mon.0) 440 : cluster [DBG] osdmap e45: 8 total, 7 up, 8 in 2024-09-17T09:20:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:11.313768+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:11.728475+0000 mon.b (mon.2) 20 : audit [DBG] from='client.? 172.21.15.103:0/807844062' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:20:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:10.566133+0000 mgr.y (mgr.14144) 136 : cluster [DBG] pgmap v127: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:11.308311+0000 mon.b (mon.2) 19 : audit [INF] from='client.? 172.21.15.103:0/3520111888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5313bab3-7c30-497b-9524-a3c63851c73a"}]: dispatch 2024-09-17T09:20:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:11.308756+0000 mon.a (mon.0) 438 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5313bab3-7c30-497b-9524-a3c63851c73a"}]: dispatch 2024-09-17T09:20:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:11.313517+0000 mon.a (mon.0) 439 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5313bab3-7c30-497b-9524-a3c63851c73a"}]': finished 2024-09-17T09:20:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:11.313617+0000 mon.a (mon.0) 440 : cluster [DBG] osdmap e45: 8 total, 7 up, 8 in 2024-09-17T09:20:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:11.313768+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:11.728475+0000 mon.b (mon.2) 20 : audit [DBG] from='client.? 172.21.15.103:0/807844062' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T09:20:14.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:12.566491+0000 mgr.y (mgr.14144) 137 : cluster [DBG] pgmap v129: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:14.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:12.566491+0000 mgr.y (mgr.14144) 137 : cluster [DBG] pgmap v129: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:14.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:12.566491+0000 mgr.y (mgr.14144) 137 : cluster [DBG] pgmap v129: 1 pgs: 1 peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:16.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:14.566887+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:16.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:14.566887+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:16.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:14.566887+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:18.063 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:16.567270+0000 mgr.y (mgr.14144) 139 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:18.064 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:17.173212+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T09:20:18.064 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:17.173965+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:20:18.064 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:20:17.174614+0000 mgr.y (mgr.14144) 140 : cephadm [INF] Deploying daemon osd.7 on smithi103 2024-09-17T09:20:18.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:16.567270+0000 mgr.y (mgr.14144) 139 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:18.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:17.173212+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T09:20:18.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:17.173965+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:20:18.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:20:17.174614+0000 mgr.y (mgr.14144) 140 : cephadm [INF] Deploying daemon osd.7 on smithi103 2024-09-17T09:20:18.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:16.567270+0000 mgr.y (mgr.14144) 139 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:18.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:17.173212+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T09:20:18.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:17.173965+0000 mon.a (mon.0) 443 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:20:18.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:20:17.174614+0000 mgr.y (mgr.14144) 140 : cephadm [INF] Deploying daemon osd.7 on smithi103 2024-09-17T09:20:20.190 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:18.567729+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:20.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:18.567729+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:20.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:18.567729+0000 mgr.y (mgr.14144) 141 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:20.719 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 7 on host 'smithi103' 2024-09-17T09:20:21.297 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:20.706562+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:21.297 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:20.711163+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:21.297 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:20.716026+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:21.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:20.706562+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:21.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:20.711163+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:21.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:20.716026+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:21.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:20.706562+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:21.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:20.711163+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:21.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:20.716026+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:21.320 DEBUG:teuthology.orchestra.run.smithi103:osd.7> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.7.service 2024-09-17T09:20:21.323 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-17T09:20:21.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-17T09:20:22.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:20.568221+0000 mgr.y (mgr.14144) 142 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:22.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:20.568221+0000 mgr.y (mgr.14144) 142 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:22.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:20.568221+0000 mgr.y (mgr.14144) 142 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:23.375 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:20:23.466 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:22.247406+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:23.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:22.247406+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:23.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:22.247406+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:24.005 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:20:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.7[33757]: debug 2024-09-17T09:20:23.818+0000 7f03c2bda080 -1 osd.7 0 log_to_monitors {default=true} 2024-09-17T09:20:24.131 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":45,"num_osds":8,"num_up_osds":7,"osd_up_since":1726564807,"num_in_osds":8,"osd_in_since":1726564811,"num_remapped_pgs":0} 2024-09-17T09:20:24.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:22.568734+0000 mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:24.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:23.374736+0000 mon.a (mon.0) 448 : audit [DBG] from='client.? 172.21.15.29:0/3787952353' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T09:20:24.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:23.822688+0000 mon.b (mon.2) 21 : audit [INF] from='osd.7 [v2:172.21.15.103:6824/1116463790,v1:172.21.15.103:6825/1116463790]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:20:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:23.823038+0000 mon.a (mon.0) 449 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:20:24.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:22.568734+0000 mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:24.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:23.374736+0000 mon.a (mon.0) 448 : audit [DBG] from='client.? 172.21.15.29:0/3787952353' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T09:20:24.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:23.822688+0000 mon.b (mon.2) 21 : audit [INF] from='osd.7 [v2:172.21.15.103:6824/1116463790,v1:172.21.15.103:6825/1116463790]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:20:24.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:23.823038+0000 mon.a (mon.0) 449 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:20:24.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:22.568734+0000 mgr.y (mgr.14144) 143 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:24.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:23.374736+0000 mon.a (mon.0) 448 : audit [DBG] from='client.? 172.21.15.29:0/3787952353' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T09:20:24.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:23.822688+0000 mon.b (mon.2) 21 : audit [INF] from='osd.7 [v2:172.21.15.103:6824/1116463790,v1:172.21.15.103:6825/1116463790]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:20:24.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:23.823038+0000 mon.a (mon.0) 449 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:20:25.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-17T09:20:25.445 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:24.257065+0000 mon.a (mon.0) 450 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T09:20:25.446 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:24.257162+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2024-09-17T09:20:25.446 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:24.257354+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:25.446 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:24.257727+0000 mon.b (mon.2) 22 : audit [INF] from='osd.7 [v2:172.21.15.103:6824/1116463790,v1:172.21.15.103:6825/1116463790]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:20:25.446 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:24.258022+0000 mon.a (mon.0) 453 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:20:25.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:24.257065+0000 mon.a (mon.0) 450 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T09:20:25.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:24.257162+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2024-09-17T09:20:25.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:24.257354+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:25.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:24.257727+0000 mon.b (mon.2) 22 : audit [INF] from='osd.7 [v2:172.21.15.103:6824/1116463790,v1:172.21.15.103:6825/1116463790]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:20:25.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:24.258022+0000 mon.a (mon.0) 453 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:20:25.755 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:20:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.7[33757]: debug 2024-09-17T09:20:25.263+0000 7f03b914b700 -1 osd.7 0 waiting for initial osdmap 2024-09-17T09:20:25.755 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:20:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.7[33757]: debug 2024-09-17T09:20:25.270+0000 7f03b38c3700 -1 osd.7 47 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-09-17T09:20:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:24.257065+0000 mon.a (mon.0) 450 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T09:20:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:24.257162+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e46: 8 total, 7 up, 8 in 2024-09-17T09:20:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:24.257354+0000 mon.a (mon.0) 452 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:24.257727+0000 mon.b (mon.2) 22 : audit [INF] from='osd.7 [v2:172.21.15.103:6824/1116463790,v1:172.21.15.103:6825/1116463790]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:20:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:24.258022+0000 mon.a (mon.0) 453 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:20:26.542 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:24.569183+0000 mgr.y (mgr.14144) 144 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:26.542 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:25.261877+0000 mon.a (mon.0) 454 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-09-17T09:20:26.542 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:25.262006+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2024-09-17T09:20:26.542 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:25.262298+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:26.542 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:25.265387+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:26.542 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:25.502623+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:26.542 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:25.505782+0000 mon.a (mon.0) 459 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:26.542 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:25.507095+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:26.543 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:25.741098+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:26.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:24.569183+0000 mgr.y (mgr.14144) 144 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:26.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:25.261877+0000 mon.a (mon.0) 454 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-09-17T09:20:26.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:25.262006+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2024-09-17T09:20:26.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:25.262298+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:26.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:25.265387+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:26.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:25.502623+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:26.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:25.505782+0000 mon.a (mon.0) 459 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:26.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:25.507095+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:26.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:25.741098+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:26.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:24.569183+0000 mgr.y (mgr.14144) 144 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:25.261877+0000 mon.a (mon.0) 454 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-09-17T09:20:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:25.262006+0000 mon.a (mon.0) 455 : cluster [DBG] osdmap e47: 8 total, 7 up, 8 in 2024-09-17T09:20:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:25.262298+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:25.265387+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:25.502623+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:25.505782+0000 mon.a (mon.0) 459 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:25.507095+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:25.741098+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:27.103 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:20:27.448 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:24.863335+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:20:27.449 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:24.863431+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:20:27.449 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:20:25.498073+0000 mgr.y (mgr.14144) 145 : cephadm [INF] Detected new or changed devices on smithi103 2024-09-17T09:20:27.449 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:26.266803+0000 mon.a (mon.0) 462 : cluster [INF] osd.7 [v2:172.21.15.103:6824/1116463790,v1:172.21.15.103:6825/1116463790] boot 2024-09-17T09:20:27.449 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:26.266850+0000 mon.a (mon.0) 463 : cluster [DBG] osdmap e48: 8 total, 8 up, 8 in 2024-09-17T09:20:27.449 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:26.267020+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:27.449 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:27.101563+0000 mon.a (mon.0) 465 : audit [DBG] from='client.? 172.21.15.29:0/1999894950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T09:20:27.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:24.863335+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:20:27.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:24.863431+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:20:27.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:20:25.498073+0000 mgr.y (mgr.14144) 145 : cephadm [INF] Detected new or changed devices on smithi103 2024-09-17T09:20:27.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:26.266803+0000 mon.a (mon.0) 462 : cluster [INF] osd.7 [v2:172.21.15.103:6824/1116463790,v1:172.21.15.103:6825/1116463790] boot 2024-09-17T09:20:27.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:26.266850+0000 mon.a (mon.0) 463 : cluster [DBG] osdmap e48: 8 total, 8 up, 8 in 2024-09-17T09:20:27.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:26.267020+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:27.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:27.101563+0000 mon.a (mon.0) 465 : audit [DBG] from='client.? 172.21.15.29:0/1999894950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T09:20:27.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:24.863335+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-17T09:20:27.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:24.863431+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-17T09:20:27.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:20:25.498073+0000 mgr.y (mgr.14144) 145 : cephadm [INF] Detected new or changed devices on smithi103 2024-09-17T09:20:27.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:26.266803+0000 mon.a (mon.0) 462 : cluster [INF] osd.7 [v2:172.21.15.103:6824/1116463790,v1:172.21.15.103:6825/1116463790] boot 2024-09-17T09:20:27.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:26.266850+0000 mon.a (mon.0) 463 : cluster [DBG] osdmap e48: 8 total, 8 up, 8 in 2024-09-17T09:20:27.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:26.267020+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:20:27.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:27.101563+0000 mon.a (mon.0) 465 : audit [DBG] from='client.? 172.21.15.29:0/1999894950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T09:20:27.733 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":48,"num_osds":8,"num_up_osds":8,"osd_up_since":1726564826,"num_in_osds":8,"osd_in_since":1726564811,"num_remapped_pgs":1} 2024-09-17T09:20:27.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T09:20:27.939 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:20:28.413 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:26.569663+0000 mgr.y (mgr.14144) 146 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:28.413 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:27.270676+0000 mon.a (mon.0) 466 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2024-09-17T09:20:28.413 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:26.569663+0000 mgr.y (mgr.14144) 146 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:28.413 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:27.270676+0000 mon.a (mon.0) 466 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2024-09-17T09:20:28.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:26.569663+0000 mgr.y (mgr.14144) 146 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2024-09-17T09:20:28.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:27.270676+0000 mon.a (mon.0) 466 : cluster [DBG] osdmap e49: 8 total, 8 up, 8 in 2024-09-17T09:20:29.375 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:28.271288+0000 mon.a (mon.0) 467 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2024-09-17T09:20:29.375 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:28.271288+0000 mon.a (mon.0) 467 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2024-09-17T09:20:29.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:28.271288+0000 mon.a (mon.0) 467 : cluster [DBG] osdmap e50: 8 total, 8 up, 8 in 2024-09-17T09:20:29.783 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:20:29.783 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"fsid":"69135c5a-74d5-11ef-bceb-c7b262605968","created":"2024-09-17T09:16:21.661201+0000","modified":"2024-09-17T09:20:28.267599+0000","last_up_change":"2024-09-17T09:20:26.259147+0000","last_in_change":"2024-09-17T09:20:11.309413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-09-17T09:18:55.536725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"97f6429c-ffd4-4aaf-bcf3-61a379d19953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3135009490},{"type":"v1","addr":"172.21.15.29:6803","nonce":3135009490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3135009490},{"type":"v1","addr":"172.21.15.29:6805","nonce":3135009490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3135009490},{"type":"v1","addr":"172.21.15.29:6809","nonce":3135009490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3135009490},{"type":"v1","addr":"172.21.15.29:6807","nonce":3135009490}]},"public_addr":"172.21.15.29:6803/3135009490","cluster_addr":"172.21.15.29:6805/3135009490","heartbeat_back_addr":"172.21.15.29:6809/3135009490","heartbeat_front_addr":"172.21.15.29:6807/3135009490","state":["exists","up"]},{"osd":1,"uuid":"ec88dcae-85cf-4f6c-8f06-c2d891eaf2c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1986066162},{"type":"v1","addr":"172.21.15.29:6811","nonce":1986066162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1986066162},{"type":"v1","addr":"172.21.15.29:6813","nonce":1986066162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1986066162},{"type":"v1","addr":"172.21.15.29:6817","nonce":1986066162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1986066162},{"type":"v1","addr":"172.21.15.29:6815","nonce":1986066162}]},"public_addr":"172.21.15.29:6811/1986066162","cluster_addr":"172.21.15.29:6813/1986066162","heartbeat_back_addr":"172.21.15.29:6817/1986066162","heartbeat_front_addr":"172.21.15.29:6815/1986066162","state":["exists","up"]},{"osd":2,"uuid":"dfb3bfde-d7f5-4570-b530-405f354cbb9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":4166454918},{"type":"v1","addr":"172.21.15.29:6819","nonce":4166454918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":4166454918},{"type":"v1","addr":"172.21.15.29:6821","nonce":4166454918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":4166454918},{"type":"v1","addr":"172.21.15.29:6825","nonce":4166454918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4166454918},{"type":"v1","addr":"172.21.15.29:6823","nonce":4166454918}]},"public_addr":"172.21.15.29:6819/4166454918","cluster_addr":"172.21.15.29:6821/4166454918","heartbeat_back_addr":"172.21.15.29:6825/4166454918","heartbeat_front_addr":"172.21.15.29:6823/4166454918","state":["exists","up"]},{"osd":3,"uuid":"8ad1a887-9ac2-42eb-a655-e7eda6e5c26e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2996228319},{"type":"v1","addr":"172.21.15.29:6827","nonce":2996228319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2996228319},{"type":"v1","addr":"172.21.15.29:6829","nonce":2996228319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2996228319},{"type":"v1","addr":"172.21.15.29:6833","nonce":2996228319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2996228319},{"type":"v1","addr":"172.21.15.29:6831","nonce":2996228319}]},"public_addr":"172.21.15.29:6827/2996228319","cluster_addr":"172.21.15.29:6829/2996228319","heartbeat_back_addr":"172.21.15.29:6833/2996228319","heartbeat_front_addr":"172.21.15.29:6831/2996228319","state":["exists","up"]},{"osd":4,"uuid":"f9b494dd-fa97-426e-9096-bfc1625f5826","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1076748350},{"type":"v1","addr":"172.21.15.103:6801","nonce":1076748350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1076748350},{"type":"v1","addr":"172.21.15.103:6803","nonce":1076748350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1076748350},{"type":"v1","addr":"172.21.15.103:6807","nonce":1076748350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1076748350},{"type":"v1","addr":"172.21.15.103:6805","nonce":1076748350}]},"public_addr":"172.21.15.103:6801/1076748350","cluster_addr":"172.21.15.103:6803/1076748350","heartbeat_back_addr":"172.21.15.103:6807/1076748350","heartbeat_front_addr":"172.21.15.103:6805/1076748350","state":["exists","up"]},{"osd":5,"uuid":"78565831-70c9-4dc9-85e2-06accbc9a1bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":902800869},{"type":"v1","addr":"172.21.15.103:6809","nonce":902800869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":902800869},{"type":"v1","addr":"172.21.15.103:6811","nonce":902800869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":902800869},{"type":"v1","addr":"172.21.15.103:6815","nonce":902800869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":902800869},{"type":"v1","addr":"172.21.15.103:6813","nonce":902800869}]},"public_addr":"172.21.15.103:6809/902800869","cluster_addr":"172.21.15.103:6811/902800869","heartbeat_back_addr":"172.21.15.103:6815/902800869","heartbeat_front_addr":"172.21.15.103:6813/902800869","state":["exists","up"]},{"osd":6,"uuid":"a3c84571-a2b3-48c3-ad16-71b144e8e332","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":3547403473},{"type":"v1","addr":"172.21.15.103:6817","nonce":3547403473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":3547403473},{"type":"v1","addr":"172.21.15.103:6819","nonce":3547403473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":3547403473},{"type":"v1","addr":"172.21.15.103:6823","nonce":3547403473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":3547403473},{"type":"v1","addr":"172.21.15.103:6821","nonce":3547403473}]},"public_addr":"172.21.15.103:6817/3547403473","cluster_addr":"172.21.15.103:6819/3547403473","heartbeat_back_addr":"172.21.15.103:6823/3547403473","heartbeat_front_addr":"172.21.15.103:6821/3547403473","state":["exists","up"]},{"osd":7,"uuid":"5313bab3-7c30-497b-9524-a3c63851c73a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1116463790},{"type":"v1","addr":"172.21.15.103:6825","nonce":1116463790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1116463790},{"type":"v1","addr":"172.21.15.103:6827","nonce":1116463790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1116463790},{"type":"v1","addr":"172.21.15.103:6831","nonce":1116463790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1116463790},{"type":"v1","addr":"172.21.15.103:6829","nonce":1116463790}]},"public_addr":"172.21.15.103:6825/1116463790","cluster_addr":"172.21.15.103:6827/1116463790","heartbeat_back_addr":"172.21.15.103:6831/1116463790","heartbeat_front_addr":"172.21.15.103:6829/1116463790","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:18:14.165854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:18:33.512550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:18:53.278038+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:19:12.962437+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:19:28.981168+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:19:47.362186+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:20:05.885758+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:20:24.863434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1475002192":"2024-09-18T09:17:04.481029+0000","172.21.15.29:0/225600821":"2024-09-18T09:16:45.142217+0000","172.21.15.29:6800/629296380":"2024-09-18T09:16:45.142217+0000","172.21.15.29:6801/395014989":"2024-09-18T09:17:04.481029+0000","172.21.15.29:6801/629296380":"2024-09-18T09:16:45.142217+0000","172.21.15.29:6800/395014989":"2024-09-18T09:17:04.481029+0000","172.21.15.29:0/2731392062":"2024-09-18T09:17:04.481029+0000","172.21.15.29:0/2438855422":"2024-09-18T09:16:45.142217+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T09:20:30.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:28.570162+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v142: 1 pgs: 1 remapped+peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:30.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:29.782208+0000 mon.a (mon.0) 468 : audit [DBG] from='client.? 172.21.15.29:0/2202672105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:20:30.362 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:28.570162+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v142: 1 pgs: 1 remapped+peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:30.362 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:29.782208+0000 mon.a (mon.0) 468 : audit [DBG] from='client.? 172.21.15.29:0/2202672105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:20:30.386 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-09-17T09:18:55.536725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2024-09-17T09:20:30.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd pool get device_health_metrics pg_num 2024-09-17T09:20:30.590 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:20:30.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:28.570162+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v142: 1 pgs: 1 remapped+peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:30.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:29.782208+0000 mon.a (mon.0) 468 : audit [DBG] from='client.? 172.21.15.29:0/2202672105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:20:31.617 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:31.278826+0000 mon.a (mon.0) 469 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-17T09:20:31.617 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:31.278826+0000 mon.a (mon.0) 469 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-17T09:20:31.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:31.278826+0000 mon.a (mon.0) 469 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-17T09:20:32.274 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-09-17T09:20:32.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:30.570485+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v143: 1 pgs: 1 remapped+peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:32.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:32.273677+0000 mon.a (mon.0) 470 : audit [DBG] from='client.? 172.21.15.29:0/3092893586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-09-17T09:20:32.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:30.570485+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v143: 1 pgs: 1 remapped+peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:32.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:32.273677+0000 mon.a (mon.0) 470 : audit [DBG] from='client.? 172.21.15.29:0/3092893586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-09-17T09:20:32.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:30.570485+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v143: 1 pgs: 1 remapped+peering; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:32.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:32.273677+0000 mon.a (mon.0) 470 : audit [DBG] from='client.? 172.21.15.29:0/3092893586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-09-17T09:20:32.850 INFO:tasks.cephadm:Adding prometheus.a on smithi103 2024-09-17T09:20:32.850 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi103=a' 2024-09-17T09:20:34.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:32.570941+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:34.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:33.332273+0000 mon.a (mon.0) 471 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-17T09:20:34.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:33.332335+0000 mon.a (mon.0) 472 : cluster [INF] Cluster is now healthy 2024-09-17T09:20:34.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:32.570941+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:34.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:33.332273+0000 mon.a (mon.0) 471 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-17T09:20:34.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:33.332335+0000 mon.a (mon.0) 472 : cluster [INF] Cluster is now healthy 2024-09-17T09:20:34.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:32.570941+0000 mgr.y (mgr.14144) 149 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:34.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:33.332273+0000 mon.a (mon.0) 471 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-17T09:20:34.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:33.332335+0000 mon.a (mon.0) 472 : cluster [INF] Cluster is now healthy 2024-09-17T09:20:34.651 INFO:teuthology.orchestra.run.smithi103.stdout:Scheduled prometheus update... 2024-09-17T09:20:35.323 DEBUG:teuthology.orchestra.run.smithi103:prometheus.a> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-17T09:20:35.326 INFO:tasks.cephadm:Adding node-exporter.a on smithi029 2024-09-17T09:20:35.326 INFO:tasks.cephadm:Adding node-exporter.b on smithi103 2024-09-17T09:20:35.326 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi029=a;smithi103=b' 2024-09-17T09:20:35.947 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:34.571288+0000 mgr.y (mgr.14144) 150 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:35.947 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:34.645708+0000 mgr.y (mgr.14144) 151 : audit [DBG] from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi103=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:35.947 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:20:34.646430+0000 mgr.y (mgr.14144) 152 : cephadm [INF] Saving service prometheus spec with placement smithi103=a;count:1 2024-09-17T09:20:35.947 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:34.650366+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:35.947 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:34.651121+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:35.947 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:34.866783+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:35.947 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:20:34.877978+0000 mgr.y (mgr.14144) 153 : cephadm [INF] Deploying daemon prometheus.a on smithi103 2024-09-17T09:20:36.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:34.571288+0000 mgr.y (mgr.14144) 150 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:36.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:34.645708+0000 mgr.y (mgr.14144) 151 : audit [DBG] from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi103=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:36.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:20:34.646430+0000 mgr.y (mgr.14144) 152 : cephadm [INF] Saving service prometheus spec with placement smithi103=a;count:1 2024-09-17T09:20:36.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:34.650366+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:36.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:34.651121+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:36.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:34.866783+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:36.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:20:34.877978+0000 mgr.y (mgr.14144) 153 : cephadm [INF] Deploying daemon prometheus.a on smithi103 2024-09-17T09:20:36.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:34.571288+0000 mgr.y (mgr.14144) 150 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:36.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:34.645708+0000 mgr.y (mgr.14144) 151 : audit [DBG] from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi103=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:36.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:20:34.646430+0000 mgr.y (mgr.14144) 152 : cephadm [INF] Saving service prometheus spec with placement smithi103=a;count:1 2024-09-17T09:20:36.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:34.650366+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:36.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:34.651121+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:36.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:34.866783+0000 mon.a (mon.0) 475 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:36.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:20:34.877978+0000 mgr.y (mgr.14144) 153 : cephadm [INF] Deploying daemon prometheus.a on smithi103 2024-09-17T09:20:37.152 INFO:teuthology.orchestra.run.smithi103.stdout:Scheduled node-exporter update... 2024-09-17T09:20:37.731 DEBUG:teuthology.orchestra.run.smithi029:node-exporter.a> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-17T09:20:37.734 DEBUG:teuthology.orchestra.run.smithi103:node-exporter.b> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-17T09:20:37.737 INFO:tasks.cephadm:Adding alertmanager.a on smithi029 2024-09-17T09:20:37.737 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi029=a' 2024-09-17T09:20:38.318 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:36.571664+0000 mgr.y (mgr.14144) 154 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:38.318 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:37.145957+0000 mgr.y (mgr.14144) 155 : audit [DBG] from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi029=a;smithi103=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:38.319 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:20:37.146784+0000 mgr.y (mgr.14144) 156 : cephadm [INF] Saving service node-exporter spec with placement smithi029=a;smithi103=b;count:2 2024-09-17T09:20:38.319 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:37.150713+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:38.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:36.571664+0000 mgr.y (mgr.14144) 154 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:38.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:37.145957+0000 mgr.y (mgr.14144) 155 : audit [DBG] from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi029=a;smithi103=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:38.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:20:37.146784+0000 mgr.y (mgr.14144) 156 : cephadm [INF] Saving service node-exporter spec with placement smithi029=a;smithi103=b;count:2 2024-09-17T09:20:38.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:37.150713+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:38.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:36.571664+0000 mgr.y (mgr.14144) 154 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:38.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:37.145957+0000 mgr.y (mgr.14144) 155 : audit [DBG] from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi029=a;smithi103=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:38.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:20:37.146784+0000 mgr.y (mgr.14144) 156 : cephadm [INF] Saving service node-exporter spec with placement smithi029=a;smithi103=b;count:2 2024-09-17T09:20:38.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:37.150713+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:39.696 INFO:teuthology.orchestra.run.smithi103.stdout:Scheduled alertmanager update... 2024-09-17T09:20:40.362 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:38.571997+0000 mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:40.363 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:39.695095+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:40.385 DEBUG:teuthology.orchestra.run.smithi029:alertmanager.a> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-17T09:20:40.389 INFO:tasks.cephadm:Adding grafana.a on smithi103 2024-09-17T09:20:40.389 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi103=a' 2024-09-17T09:20:40.413 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:38.571997+0000 mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:40.413 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:39.695095+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:40.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:38.571997+0000 mgr.y (mgr.14144) 157 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:40.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:39.695095+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:41.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:39.690557+0000 mgr.y (mgr.14144) 158 : audit [DBG] from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi029=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:41.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:20:39.691314+0000 mgr.y (mgr.14144) 159 : cephadm [INF] Saving service alertmanager spec with placement smithi029=a;count:1 2024-09-17T09:20:41.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:39.690557+0000 mgr.y (mgr.14144) 158 : audit [DBG] from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi029=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:41.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:20:39.691314+0000 mgr.y (mgr.14144) 159 : cephadm [INF] Saving service alertmanager spec with placement smithi029=a;count:1 2024-09-17T09:20:41.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:39.690557+0000 mgr.y (mgr.14144) 158 : audit [DBG] from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi029=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:41.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:20:39.691314+0000 mgr.y (mgr.14144) 159 : cephadm [INF] Saving service alertmanager spec with placement smithi029=a;count:1 2024-09-17T09:20:42.239 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:42 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:40.572409+0000 mgr.y (mgr.14144) 160 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:42.384 INFO:teuthology.orchestra.run.smithi103.stdout:Scheduled grafana update... 2024-09-17T09:20:42.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:40.572409+0000 mgr.y (mgr.14144) 160 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:42.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:40.572409+0000 mgr.y (mgr.14144) 160 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:43.329 DEBUG:teuthology.orchestra.run.smithi103:grafana.a> sudo journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@grafana.a.service 2024-09-17T09:20:43.331 INFO:tasks.cephadm:Setting up client nodes... 2024-09-17T09:20:43.332 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T09:20:43.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:42.379176+0000 mgr.y (mgr.14144) 161 : audit [DBG] from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi103=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:43.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:20:42.379919+0000 mgr.y (mgr.14144) 162 : cephadm [INF] Saving service grafana spec with placement smithi103=a;count:1 2024-09-17T09:20:43.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:42.383796+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:43.627 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:42.379176+0000 mgr.y (mgr.14144) 161 : audit [DBG] from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi103=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:43.627 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:20:42.379919+0000 mgr.y (mgr.14144) 162 : cephadm [INF] Saving service grafana spec with placement smithi103=a;count:1 2024-09-17T09:20:43.628 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:42.383796+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:43.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:42.379176+0000 mgr.y (mgr.14144) 161 : audit [DBG] from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi103=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:20:43.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:20:42.379919+0000 mgr.y (mgr.14144) 162 : cephadm [INF] Saving service grafana spec with placement smithi103=a;count:1 2024-09-17T09:20:43.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:42.383796+0000 mon.a (mon.0) 478 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:44.656 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:42.572953+0000 mgr.y (mgr.14144) 163 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:44.760 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:42.572953+0000 mgr.y (mgr.14144) 163 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:44.760 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:42.572953+0000 mgr.y (mgr.14144) 163 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:45.361 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-09-17T09:20:45.361 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDtSelmRMFTFRAA6Lc0PZ+tnqgWmpx/P4nPqA== 2024-09-17T09:20:45.669 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:45.357641+0000 mon.a (mon.0) 479 : audit [INF] from='client.? 172.21.15.29:0/1835894809' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:20:45.669 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:45.360849+0000 mon.a (mon.0) 480 : audit [INF] from='client.? 172.21.15.29:0/1835894809' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T09:20:45.717 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:45.357641+0000 mon.a (mon.0) 479 : audit [INF] from='client.? 172.21.15.29:0/1835894809' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:20:45.718 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:45.360849+0000 mon.a (mon.0) 480 : audit [INF] from='client.? 172.21.15.29:0/1835894809' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T09:20:45.718 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:45.357641+0000 mon.a (mon.0) 479 : audit [INF] from='client.? 172.21.15.29:0/1835894809' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:20:45.718 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:45.360849+0000 mon.a (mon.0) 480 : audit [INF] from='client.? 172.21.15.29:0/1835894809' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T09:20:46.034 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-17T09:20:46.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-17T09:20:46.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T09:20:46.078 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T09:20:46.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:44.573351+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:46.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:44.573351+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:46.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:44.573351+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:48.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:46.573858+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:48.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:46.573858+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:48.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:46.573858+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:50.142 INFO:teuthology.orchestra.run.smithi103.stdout:[client.1] 2024-09-17T09:20:50.142 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQDySelm5HE4CBAA8Q3QTwGX+xFXPSZ3lE4uwQ== 2024-09-17T09:20:50.594 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:48.574217+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:50.595 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:50.137140+0000 mon.b (mon.2) 23 : audit [INF] from='client.? 172.21.15.103:0/2911046783' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:20:50.595 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:50.137739+0000 mon.a (mon.0) 481 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:20:50.595 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:50.141438+0000 mon.a (mon.0) 482 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T09:20:50.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:48.574217+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:50.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:50.137140+0000 mon.b (mon.2) 23 : audit [INF] from='client.? 172.21.15.103:0/2911046783' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:20:50.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:50.137739+0000 mon.a (mon.0) 481 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:20:50.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:50.141438+0000 mon.a (mon.0) 482 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T09:20:50.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:48.574217+0000 mgr.y (mgr.14144) 166 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:50.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:50.137140+0000 mon.b (mon.2) 23 : audit [INF] from='client.? 172.21.15.103:0/2911046783' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:20:50.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:50.137739+0000 mon.a (mon.0) 481 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T09:20:50.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:50.141438+0000 mon.a (mon.0) 482 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T09:20:50.892 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-09-17T09:20:50.892 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-17T09:20:50.892 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-17T09:20:50.930 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T09:20:50.930 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-17T09:20:50.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-17T09:20:51.147 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:20:52.487 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:50.574611+0000 mgr.y (mgr.14144) 167 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:52.487 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:50.574611+0000 mgr.y (mgr.14144) 167 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:52.689 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:50.574611+0000 mgr.y (mgr.14144) 167 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:52.905 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:20:53.255 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:20:53.533 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":14,"active_gid":14144,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":536255971},{"type":"v1","addr":"172.21.15.29:6801","nonce":536255971}]},"active_addr":"172.21.15.29:6801/536255971","active_change":"2024-09-17T09:17:04.481341+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":34109,"name":"x","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi029.front.sepia.ceph.com:8443/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1161843867}]},{"addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":688662900}]}]}} 2024-09-17T09:20:53.537 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-17T09:20:53.537 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-17T09:20:53.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T09:20:53.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:52.900090+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.29:0/2994017261' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T09:20:53.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:52.900090+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.29:0/2994017261' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T09:20:53.635 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:52.900090+0000 mon.c (mon.1) 11 : audit [DBG] from='client.? 172.21.15.29:0/2994017261' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T09:20:53.635 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 podman[35767]: 2024-09-17 09:20:53.25629649 +0000 UTC m=+0.016579123 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:20:53.635 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 podman[35767]: 2024-09-17 09:20:53.424651895 +0000 UTC m=+0.184934517 container create b995939d97369b77770a41cb6396bc3d9487fd3f97589bb3349515f4db506ced (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:20:53.635 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 podman[35767]: 2024-09-17 09:20:53.634637734 +0000 UTC m=+0.394920365 container init b995939d97369b77770a41cb6396bc3d9487fd3f97589bb3349515f4db506ced (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:20:53.740 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:20:54.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 podman[35767]: 2024-09-17 09:20:53.637522687 +0000 UTC m=+0.397805316 container start b995939d97369b77770a41cb6396bc3d9487fd3f97589bb3349515f4db506ced (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:20:54.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.667Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:20:54.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.667Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:20:54.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.667Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:20:54.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.667Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:20:54.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.667Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:20:54.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.667Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:20:54.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.668Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:20:54.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.668Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:20:54.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.671Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:20:54.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.671Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-17T09:20:54.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.672Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=905.148µs 2024-09-17T09:20:54.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.672Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:20:54.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.672Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:20:54.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.672Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:20:54.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.674Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:20:54.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:20:53.674Z caller=main.go:646 msg="Server is ready to receive web requests." 2024-09-17T09:20:54.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 bash[35767]: b995939d97369b77770a41cb6396bc3d9487fd3f97589bb3349515f4db506ced 2024-09-17T09:20:54.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:20:53 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:20:54.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:52.574983+0000 mgr.y (mgr.14144) 168 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:54.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:53.760447+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:54.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:53.761151+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:54.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:52.574983+0000 mgr.y (mgr.14144) 168 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:54.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:53.760447+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:54.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:53.761151+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:54.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:52.574983+0000 mgr.y (mgr.14144) 168 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:54.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:53.760447+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:54.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:53.761151+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:20:55.466 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:20:55.466 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"fsid":"69135c5a-74d5-11ef-bceb-c7b262605968","created":"2024-09-17T09:16:21.661201+0000","modified":"2024-09-17T09:20:28.267599+0000","last_up_change":"2024-09-17T09:20:26.259147+0000","last_in_change":"2024-09-17T09:20:11.309413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-09-17T09:18:55.536725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"97f6429c-ffd4-4aaf-bcf3-61a379d19953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3135009490},{"type":"v1","addr":"172.21.15.29:6803","nonce":3135009490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3135009490},{"type":"v1","addr":"172.21.15.29:6805","nonce":3135009490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3135009490},{"type":"v1","addr":"172.21.15.29:6809","nonce":3135009490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3135009490},{"type":"v1","addr":"172.21.15.29:6807","nonce":3135009490}]},"public_addr":"172.21.15.29:6803/3135009490","cluster_addr":"172.21.15.29:6805/3135009490","heartbeat_back_addr":"172.21.15.29:6809/3135009490","heartbeat_front_addr":"172.21.15.29:6807/3135009490","state":["exists","up"]},{"osd":1,"uuid":"ec88dcae-85cf-4f6c-8f06-c2d891eaf2c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1986066162},{"type":"v1","addr":"172.21.15.29:6811","nonce":1986066162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1986066162},{"type":"v1","addr":"172.21.15.29:6813","nonce":1986066162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1986066162},{"type":"v1","addr":"172.21.15.29:6817","nonce":1986066162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1986066162},{"type":"v1","addr":"172.21.15.29:6815","nonce":1986066162}]},"public_addr":"172.21.15.29:6811/1986066162","cluster_addr":"172.21.15.29:6813/1986066162","heartbeat_back_addr":"172.21.15.29:6817/1986066162","heartbeat_front_addr":"172.21.15.29:6815/1986066162","state":["exists","up"]},{"osd":2,"uuid":"dfb3bfde-d7f5-4570-b530-405f354cbb9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":4166454918},{"type":"v1","addr":"172.21.15.29:6819","nonce":4166454918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":4166454918},{"type":"v1","addr":"172.21.15.29:6821","nonce":4166454918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":4166454918},{"type":"v1","addr":"172.21.15.29:6825","nonce":4166454918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4166454918},{"type":"v1","addr":"172.21.15.29:6823","nonce":4166454918}]},"public_addr":"172.21.15.29:6819/4166454918","cluster_addr":"172.21.15.29:6821/4166454918","heartbeat_back_addr":"172.21.15.29:6825/4166454918","heartbeat_front_addr":"172.21.15.29:6823/4166454918","state":["exists","up"]},{"osd":3,"uuid":"8ad1a887-9ac2-42eb-a655-e7eda6e5c26e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2996228319},{"type":"v1","addr":"172.21.15.29:6827","nonce":2996228319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2996228319},{"type":"v1","addr":"172.21.15.29:6829","nonce":2996228319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2996228319},{"type":"v1","addr":"172.21.15.29:6833","nonce":2996228319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2996228319},{"type":"v1","addr":"172.21.15.29:6831","nonce":2996228319}]},"public_addr":"172.21.15.29:6827/2996228319","cluster_addr":"172.21.15.29:6829/2996228319","heartbeat_back_addr":"172.21.15.29:6833/2996228319","heartbeat_front_addr":"172.21.15.29:6831/2996228319","state":["exists","up"]},{"osd":4,"uuid":"f9b494dd-fa97-426e-9096-bfc1625f5826","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1076748350},{"type":"v1","addr":"172.21.15.103:6801","nonce":1076748350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1076748350},{"type":"v1","addr":"172.21.15.103:6803","nonce":1076748350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1076748350},{"type":"v1","addr":"172.21.15.103:6807","nonce":1076748350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1076748350},{"type":"v1","addr":"172.21.15.103:6805","nonce":1076748350}]},"public_addr":"172.21.15.103:6801/1076748350","cluster_addr":"172.21.15.103:6803/1076748350","heartbeat_back_addr":"172.21.15.103:6807/1076748350","heartbeat_front_addr":"172.21.15.103:6805/1076748350","state":["exists","up"]},{"osd":5,"uuid":"78565831-70c9-4dc9-85e2-06accbc9a1bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":902800869},{"type":"v1","addr":"172.21.15.103:6809","nonce":902800869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":902800869},{"type":"v1","addr":"172.21.15.103:6811","nonce":902800869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":902800869},{"type":"v1","addr":"172.21.15.103:6815","nonce":902800869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":902800869},{"type":"v1","addr":"172.21.15.103:6813","nonce":902800869}]},"public_addr":"172.21.15.103:6809/902800869","cluster_addr":"172.21.15.103:6811/902800869","heartbeat_back_addr":"172.21.15.103:6815/902800869","heartbeat_front_addr":"172.21.15.103:6813/902800869","state":["exists","up"]},{"osd":6,"uuid":"a3c84571-a2b3-48c3-ad16-71b144e8e332","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":3547403473},{"type":"v1","addr":"172.21.15.103:6817","nonce":3547403473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":3547403473},{"type":"v1","addr":"172.21.15.103:6819","nonce":3547403473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":3547403473},{"type":"v1","addr":"172.21.15.103:6823","nonce":3547403473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":3547403473},{"type":"v1","addr":"172.21.15.103:6821","nonce":3547403473}]},"public_addr":"172.21.15.103:6817/3547403473","cluster_addr":"172.21.15.103:6819/3547403473","heartbeat_back_addr":"172.21.15.103:6823/3547403473","heartbeat_front_addr":"172.21.15.103:6821/3547403473","state":["exists","up"]},{"osd":7,"uuid":"5313bab3-7c30-497b-9524-a3c63851c73a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1116463790},{"type":"v1","addr":"172.21.15.103:6825","nonce":1116463790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1116463790},{"type":"v1","addr":"172.21.15.103:6827","nonce":1116463790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1116463790},{"type":"v1","addr":"172.21.15.103:6831","nonce":1116463790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1116463790},{"type":"v1","addr":"172.21.15.103:6829","nonce":1116463790}]},"public_addr":"172.21.15.103:6825/1116463790","cluster_addr":"172.21.15.103:6827/1116463790","heartbeat_back_addr":"172.21.15.103:6831/1116463790","heartbeat_front_addr":"172.21.15.103:6829/1116463790","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:18:14.165854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:18:33.512550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:18:53.278038+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:19:12.962437+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:19:28.981168+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:19:47.362186+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:20:05.885758+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:20:24.863434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1475002192":"2024-09-18T09:17:04.481029+0000","172.21.15.29:0/225600821":"2024-09-18T09:16:45.142217+0000","172.21.15.29:6800/629296380":"2024-09-18T09:16:45.142217+0000","172.21.15.29:6801/395014989":"2024-09-18T09:17:04.481029+0000","172.21.15.29:6801/629296380":"2024-09-18T09:16:45.142217+0000","172.21.15.29:6800/395014989":"2024-09-18T09:17:04.481029+0000","172.21.15.29:0/2731392062":"2024-09-18T09:17:04.481029+0000","172.21.15.29:0/2438855422":"2024-09-18T09:16:45.142217+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T09:20:55.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:54.575370+0000 mgr.y (mgr.14144) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:55.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:54.575370+0000 mgr.y (mgr.14144) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:55.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:54.575370+0000 mgr.y (mgr.14144) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:56.286 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-17T09:20:56.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T09:20:56.513 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:55.464770+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.29:0/260094374' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:20:56.513 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:55.638556+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:56.513 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:55.641567+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:56.513 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:20:55.642872+0000 mgr.y (mgr.14144) 170 : cephadm [INF] Deploying daemon node-exporter.a on smithi029 2024-09-17T09:20:56.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:55.464770+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.29:0/260094374' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:20:56.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:55.638556+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:56.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:55.641567+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:56.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:20:55.642872+0000 mgr.y (mgr.14144) 170 : cephadm [INF] Deploying daemon node-exporter.a on smithi029 2024-09-17T09:20:56.679 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:20:56.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:55.464770+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.29:0/260094374' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:20:56.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:55.638556+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:56.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:55.641567+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:20:56.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:20:55.642872+0000 mgr.y (mgr.14144) 170 : cephadm [INF] Deploying daemon node-exporter.a on smithi029 2024-09-17T09:20:56.797 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:20:56 smithi029 systemd[1]: Starting Ceph node-exporter.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:20:57.421 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:20:57 smithi029 bash[38575]: Trying to pull docker.io/prom/node-exporter:v0.18.1... 2024-09-17T09:20:57.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:56.575688+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:57.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:56.575688+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:57.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:56.575688+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:20:58.498 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:20:58.499 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"fsid":"69135c5a-74d5-11ef-bceb-c7b262605968","created":"2024-09-17T09:16:21.661201+0000","modified":"2024-09-17T09:20:28.267599+0000","last_up_change":"2024-09-17T09:20:26.259147+0000","last_in_change":"2024-09-17T09:20:11.309413+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-09-17T09:18:55.536725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"97f6429c-ffd4-4aaf-bcf3-61a379d19953","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3135009490},{"type":"v1","addr":"172.21.15.29:6803","nonce":3135009490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3135009490},{"type":"v1","addr":"172.21.15.29:6805","nonce":3135009490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3135009490},{"type":"v1","addr":"172.21.15.29:6809","nonce":3135009490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3135009490},{"type":"v1","addr":"172.21.15.29:6807","nonce":3135009490}]},"public_addr":"172.21.15.29:6803/3135009490","cluster_addr":"172.21.15.29:6805/3135009490","heartbeat_back_addr":"172.21.15.29:6809/3135009490","heartbeat_front_addr":"172.21.15.29:6807/3135009490","state":["exists","up"]},{"osd":1,"uuid":"ec88dcae-85cf-4f6c-8f06-c2d891eaf2c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":1986066162},{"type":"v1","addr":"172.21.15.29:6811","nonce":1986066162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":1986066162},{"type":"v1","addr":"172.21.15.29:6813","nonce":1986066162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1986066162},{"type":"v1","addr":"172.21.15.29:6817","nonce":1986066162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1986066162},{"type":"v1","addr":"172.21.15.29:6815","nonce":1986066162}]},"public_addr":"172.21.15.29:6811/1986066162","cluster_addr":"172.21.15.29:6813/1986066162","heartbeat_back_addr":"172.21.15.29:6817/1986066162","heartbeat_front_addr":"172.21.15.29:6815/1986066162","state":["exists","up"]},{"osd":2,"uuid":"dfb3bfde-d7f5-4570-b530-405f354cbb9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":4166454918},{"type":"v1","addr":"172.21.15.29:6819","nonce":4166454918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":4166454918},{"type":"v1","addr":"172.21.15.29:6821","nonce":4166454918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":4166454918},{"type":"v1","addr":"172.21.15.29:6825","nonce":4166454918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":4166454918},{"type":"v1","addr":"172.21.15.29:6823","nonce":4166454918}]},"public_addr":"172.21.15.29:6819/4166454918","cluster_addr":"172.21.15.29:6821/4166454918","heartbeat_back_addr":"172.21.15.29:6825/4166454918","heartbeat_front_addr":"172.21.15.29:6823/4166454918","state":["exists","up"]},{"osd":3,"uuid":"8ad1a887-9ac2-42eb-a655-e7eda6e5c26e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":2996228319},{"type":"v1","addr":"172.21.15.29:6827","nonce":2996228319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":2996228319},{"type":"v1","addr":"172.21.15.29:6829","nonce":2996228319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2996228319},{"type":"v1","addr":"172.21.15.29:6833","nonce":2996228319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":2996228319},{"type":"v1","addr":"172.21.15.29:6831","nonce":2996228319}]},"public_addr":"172.21.15.29:6827/2996228319","cluster_addr":"172.21.15.29:6829/2996228319","heartbeat_back_addr":"172.21.15.29:6833/2996228319","heartbeat_front_addr":"172.21.15.29:6831/2996228319","state":["exists","up"]},{"osd":4,"uuid":"f9b494dd-fa97-426e-9096-bfc1625f5826","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1076748350},{"type":"v1","addr":"172.21.15.103:6801","nonce":1076748350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":1076748350},{"type":"v1","addr":"172.21.15.103:6803","nonce":1076748350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":1076748350},{"type":"v1","addr":"172.21.15.103:6807","nonce":1076748350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":1076748350},{"type":"v1","addr":"172.21.15.103:6805","nonce":1076748350}]},"public_addr":"172.21.15.103:6801/1076748350","cluster_addr":"172.21.15.103:6803/1076748350","heartbeat_back_addr":"172.21.15.103:6807/1076748350","heartbeat_front_addr":"172.21.15.103:6805/1076748350","state":["exists","up"]},{"osd":5,"uuid":"78565831-70c9-4dc9-85e2-06accbc9a1bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":902800869},{"type":"v1","addr":"172.21.15.103:6809","nonce":902800869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":902800869},{"type":"v1","addr":"172.21.15.103:6811","nonce":902800869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":902800869},{"type":"v1","addr":"172.21.15.103:6815","nonce":902800869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":902800869},{"type":"v1","addr":"172.21.15.103:6813","nonce":902800869}]},"public_addr":"172.21.15.103:6809/902800869","cluster_addr":"172.21.15.103:6811/902800869","heartbeat_back_addr":"172.21.15.103:6815/902800869","heartbeat_front_addr":"172.21.15.103:6813/902800869","state":["exists","up"]},{"osd":6,"uuid":"a3c84571-a2b3-48c3-ad16-71b144e8e332","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":3547403473},{"type":"v1","addr":"172.21.15.103:6817","nonce":3547403473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":3547403473},{"type":"v1","addr":"172.21.15.103:6819","nonce":3547403473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":3547403473},{"type":"v1","addr":"172.21.15.103:6823","nonce":3547403473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":3547403473},{"type":"v1","addr":"172.21.15.103:6821","nonce":3547403473}]},"public_addr":"172.21.15.103:6817/3547403473","cluster_addr":"172.21.15.103:6819/3547403473","heartbeat_back_addr":"172.21.15.103:6823/3547403473","heartbeat_front_addr":"172.21.15.103:6821/3547403473","state":["exists","up"]},{"osd":7,"uuid":"5313bab3-7c30-497b-9524-a3c63851c73a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1116463790},{"type":"v1","addr":"172.21.15.103:6825","nonce":1116463790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1116463790},{"type":"v1","addr":"172.21.15.103:6827","nonce":1116463790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1116463790},{"type":"v1","addr":"172.21.15.103:6831","nonce":1116463790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1116463790},{"type":"v1","addr":"172.21.15.103:6829","nonce":1116463790}]},"public_addr":"172.21.15.103:6825/1116463790","cluster_addr":"172.21.15.103:6827/1116463790","heartbeat_back_addr":"172.21.15.103:6831/1116463790","heartbeat_front_addr":"172.21.15.103:6829/1116463790","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:18:14.165854+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:18:33.512550+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:18:53.278038+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:19:12.962437+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:19:28.981168+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:19:47.362186+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:20:05.885758+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T09:20:24.863434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1475002192":"2024-09-18T09:17:04.481029+0000","172.21.15.29:0/225600821":"2024-09-18T09:16:45.142217+0000","172.21.15.29:6800/629296380":"2024-09-18T09:16:45.142217+0000","172.21.15.29:6801/395014989":"2024-09-18T09:17:04.481029+0000","172.21.15.29:6801/629296380":"2024-09-18T09:16:45.142217+0000","172.21.15.29:6800/395014989":"2024-09-18T09:17:04.481029+0000","172.21.15.29:0/2731392062":"2024-09-18T09:17:04.481029+0000","172.21.15.29:0/2438855422":"2024-09-18T09:16:45.142217+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T09:20:59.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:20:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:20:58.497262+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.29:0/527435965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:20:59.183 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:20:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:20:58.497262+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.29:0/527435965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:20:59.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-17T09:20:59.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-17T09:20:59.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-17T09:20:59.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-17T09:20:59.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-17T09:20:59.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-17T09:20:59.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-17T09:20:59.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-17T09:20:59.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:20:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:20:58.497262+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.29:0/527435965' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T09:20:59.607 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:20:59.607 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:20:59.610 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:20:59.614 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:20:59.614 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:20:59.614 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:20:59.616 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:20:59.626 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:21:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:20:58.575957+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:20:58.575957+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:00.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:20:58.575957+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:02.268 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:00.576232+0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:02.268 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:00.576232+0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:02.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:00.576232+0000 mgr.y (mgr.14144) 173 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:03.798 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:03 smithi029 bash[38575]: Getting image source signatures 2024-09-17T09:21:03.798 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:03 smithi029 bash[38575]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2024-09-17T09:21:03.798 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:03 smithi029 bash[38575]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2024-09-17T09:21:03.798 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:03 smithi029 bash[38575]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2024-09-17T09:21:04.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:02.576616+0000 mgr.y (mgr.14144) 174 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:04.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:02.576616+0000 mgr.y (mgr.14144) 174 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:04.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:02.576616+0000 mgr.y (mgr.14144) 174 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:05.293 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:04.671160+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:21:05.294 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:04.672131+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:21:05.294 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:04.671160+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:21:05.294 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:04.672131+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:21:05.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:04.671160+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:21:05.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:04.672131+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:21:06.018 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:05 smithi029 bash[38575]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2024-09-17T09:21:06.018 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:05 smithi029 bash[38575]: Writing manifest to image destination 2024-09-17T09:21:06.270 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:04.577123+0000 mgr.y (mgr.14144) 175 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:06.270 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:04.577123+0000 mgr.y (mgr.14144) 175 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:06.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:04.577123+0000 mgr.y (mgr.14144) 175 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:06.547 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:06 smithi029 podman[38575]: 2024-09-17 09:21:06.375688001 +0000 UTC m=+9.383302506 image pull e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 docker.io/prom/node-exporter:v0.18.1 2024-09-17T09:21:08.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:06.577568+0000 mgr.y (mgr.14144) 176 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:08.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:06.577568+0000 mgr.y (mgr.14144) 176 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:08.299 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:07 smithi029 podman[38575]: 2024-09-17 09:21:07.937884248 +0000 UTC m=+10.945498733 container create 8202f7eb3e968b92e98e5ba915e53e7dd8ec0a47bd28a5ae7c9adaefa88523c3 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T09:21:08.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:06.577568+0000 mgr.y (mgr.14144) 176 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:08.929 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 podman[38575]: 2024-09-17 09:21:08.707984177 +0000 UTC m=+11.715598680 container init 8202f7eb3e968b92e98e5ba915e53e7dd8ec0a47bd28a5ae7c9adaefa88523c3 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T09:21:08.929 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 podman[38575]: 2024-09-17 09:21:08.711611486 +0000 UTC m=+11.719225977 container start 8202f7eb3e968b92e98e5ba915e53e7dd8ec0a47bd28a5ae7c9adaefa88523c3 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T09:21:08.929 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2024-09-17T09:21:08.929 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2024-09-17T09:21:08.929 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2024-09-17T09:21:08.930 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - arp" source="node_exporter.go:104" 2024-09-17T09:21:08.930 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - bcache" source="node_exporter.go:104" 2024-09-17T09:21:08.930 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - bonding" source="node_exporter.go:104" 2024-09-17T09:21:08.930 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2024-09-17T09:21:08.930 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - cpu" source="node_exporter.go:104" 2024-09-17T09:21:08.930 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2024-09-17T09:21:08.930 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2024-09-17T09:21:08.930 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - edac" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - entropy" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - filefd" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - netclass" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - netdev" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - netstat" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - nfs" source="node_exporter.go:104" 2024-09-17T09:21:08.931 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2024-09-17T09:21:08.932 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - pressure" source="node_exporter.go:104" 2024-09-17T09:21:08.932 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2024-09-17T09:21:08.932 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - stat" source="node_exporter.go:104" 2024-09-17T09:21:08.932 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - textfile" source="node_exporter.go:104" 2024-09-17T09:21:08.932 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - time" source="node_exporter.go:104" 2024-09-17T09:21:08.932 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - uname" source="node_exporter.go:104" 2024-09-17T09:21:08.932 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2024-09-17T09:21:08.932 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - xfs" source="node_exporter.go:104" 2024-09-17T09:21:08.932 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg=" - zfs" source="node_exporter.go:104" 2024-09-17T09:21:08.932 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[39358]: time="2024-09-17T09:21:08Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2024-09-17T09:21:08.932 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 bash[38575]: 8202f7eb3e968b92e98e5ba915e53e7dd8ec0a47bd28a5ae7c9adaefa88523c3 2024-09-17T09:21:08.932 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:21:08 smithi029 systemd[1]: Started Ceph node-exporter.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:21:09.755 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:09 smithi103 bash[36190]: Trying to pull docker.io/prom/node-exporter:v0.18.1... 2024-09-17T09:21:10.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:08.577984+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:10.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:08.886568+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:10.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:21:08.887247+0000 mgr.y (mgr.14144) 178 : cephadm [INF] Deploying daemon node-exporter.b on smithi103 2024-09-17T09:21:10.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:08.577984+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:10.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:08.886568+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:10.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:21:08.887247+0000 mgr.y (mgr.14144) 178 : cephadm [INF] Deploying daemon node-exporter.b on smithi103 2024-09-17T09:21:10.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:08.577984+0000 mgr.y (mgr.14144) 177 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:10.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:08.886568+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:10.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:21:08.887247+0000 mgr.y (mgr.14144) 178 : cephadm [INF] Deploying daemon node-exporter.b on smithi103 2024-09-17T09:21:11.987 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:10.578317+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:11.987 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:10.578317+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:12.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:10.578317+0000 mgr.y (mgr.14144) 179 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:13.834 INFO:teuthology.orchestra.run.smithi029.stdout:77309411359 2024-09-17T09:21:13.835 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-17T09:21:14.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:12.578755+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:14.063 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:12.578755+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:14.300 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:12.578755+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:15.004 INFO:teuthology.orchestra.run.smithi029.stdout:34359738404 2024-09-17T09:21:15.005 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-17T09:21:16.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:14.579091+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:16.159 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:14.579091+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:16.185 INFO:teuthology.orchestra.run.smithi029.stdout:55834574883 2024-09-17T09:21:16.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-17T09:21:16.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:14.579091+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:16.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:16 smithi103 bash[36190]: Getting image source signatures 2024-09-17T09:21:16.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:16 smithi103 bash[36190]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2024-09-17T09:21:16.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:16 smithi103 bash[36190]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2024-09-17T09:21:16.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:16 smithi103 bash[36190]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2024-09-17T09:21:16.801 INFO:teuthology.orchestra.run.smithi029.stdout:180388626448 2024-09-17T09:21:16.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-17T09:21:17.348 INFO:teuthology.orchestra.run.smithi029.stdout:154618822674 2024-09-17T09:21:17.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-17T09:21:17.657 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:21:17.657 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:21:17.664 INFO:teuthology.orchestra.run.smithi029.stdout:206158430219 2024-09-17T09:21:17.665 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-17T09:21:17.962 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:17 smithi103 bash[36190]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2024-09-17T09:21:17.962 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:17 smithi103 bash[36190]: Writing manifest to image destination 2024-09-17T09:21:18.240 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:16.579634+0000 mgr.y (mgr.14144) 182 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:18.240 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:17 smithi103 podman[36190]: 2024-09-17 09:21:17.955744699 +0000 UTC m=+8.431684688 image pull e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 docker.io/prom/node-exporter:v0.18.1 2024-09-17T09:21:18.240 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 podman[36190]: 2024-09-17 09:21:18.054122298 +0000 UTC m=+8.530062288 container create 0e69315832421c42e8ec61fc749374afdd7e280afafe7734c1105e40405ef391 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T09:21:18.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:16.579634+0000 mgr.y (mgr.14144) 182 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:18.248 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:16.579634+0000 mgr.y (mgr.14144) 182 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:18.505 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 podman[36190]: 2024-09-17 09:21:18.239373907 +0000 UTC m=+8.715313899 container init 0e69315832421c42e8ec61fc749374afdd7e280afafe7734c1105e40405ef391 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T09:21:18.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 podman[36190]: 2024-09-17 09:21:18.24286026 +0000 UTC m=+8.718800252 container start 0e69315832421c42e8ec61fc749374afdd7e280afafe7734c1105e40405ef391 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T09:21:18.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2024-09-17T09:21:18.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2024-09-17T09:21:18.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2024-09-17T09:21:18.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - arp" source="node_exporter.go:104" 2024-09-17T09:21:18.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - bcache" source="node_exporter.go:104" 2024-09-17T09:21:18.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - bonding" source="node_exporter.go:104" 2024-09-17T09:21:18.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2024-09-17T09:21:18.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - cpu" source="node_exporter.go:104" 2024-09-17T09:21:18.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2024-09-17T09:21:18.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2024-09-17T09:21:18.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - edac" source="node_exporter.go:104" 2024-09-17T09:21:18.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - entropy" source="node_exporter.go:104" 2024-09-17T09:21:18.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - filefd" source="node_exporter.go:104" 2024-09-17T09:21:18.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2024-09-17T09:21:18.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2024-09-17T09:21:18.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2024-09-17T09:21:18.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2024-09-17T09:21:18.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2024-09-17T09:21:18.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2024-09-17T09:21:18.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2024-09-17T09:21:18.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - netclass" source="node_exporter.go:104" 2024-09-17T09:21:18.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - netdev" source="node_exporter.go:104" 2024-09-17T09:21:18.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - netstat" source="node_exporter.go:104" 2024-09-17T09:21:18.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - nfs" source="node_exporter.go:104" 2024-09-17T09:21:18.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2024-09-17T09:21:18.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - pressure" source="node_exporter.go:104" 2024-09-17T09:21:18.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2024-09-17T09:21:18.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - stat" source="node_exporter.go:104" 2024-09-17T09:21:18.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - textfile" source="node_exporter.go:104" 2024-09-17T09:21:18.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - time" source="node_exporter.go:104" 2024-09-17T09:21:18.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - uname" source="node_exporter.go:104" 2024-09-17T09:21:18.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2024-09-17T09:21:18.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - xfs" source="node_exporter.go:104" 2024-09-17T09:21:18.509 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg=" - zfs" source="node_exporter.go:104" 2024-09-17T09:21:18.509 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[36277]: time="2024-09-17T09:21:18Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2024-09-17T09:21:18.509 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 bash[36190]: 0e69315832421c42e8ec61fc749374afdd7e280afafe7734c1105e40405ef391 2024-09-17T09:21:18.509 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:21:18 smithi103 systemd[1]: Started Ceph node-exporter.b for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:21:19.099 INFO:teuthology.orchestra.run.smithi029.stdout:128849018904 2024-09-17T09:21:19.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-17T09:21:19.505 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:21:19.506 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:21:19.506 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:21:19.507 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:21:19.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:18.396819+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:19.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:21:18.401872+0000 mgr.y (mgr.14144) 183 : cephadm [INF] Deploying daemon alertmanager.a on smithi029 2024-09-17T09:21:19.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:18.396819+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:19.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:21:18.401872+0000 mgr.y (mgr.14144) 183 : cephadm [INF] Deploying daemon alertmanager.a on smithi029 2024-09-17T09:21:19.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:18.396819+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:19.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:21:18.401872+0000 mgr.y (mgr.14144) 183 : cephadm [INF] Deploying daemon alertmanager.a on smithi029 2024-09-17T09:21:20.022 INFO:teuthology.orchestra.run.smithi029.stdout:107374182427 2024-09-17T09:21:20.022 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-17T09:21:20.273 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:21:20.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:18.580009+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:20.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:18.580009+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:20.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:18.580009+0000 mgr.y (mgr.14144) 184 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:21.339 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:21:22.663 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:20.580311+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:22.664 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:20.580311+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:22.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:20.580311+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:24.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:22.580608+0000 mgr.y (mgr.14144) 186 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:24.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:24.285003+0000 mon.a (mon.0) 491 : audit [DBG] from='client.? 172.21.15.29:0/2827676416' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T09:21:24.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:22.580608+0000 mgr.y (mgr.14144) 186 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:24.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:24.285003+0000 mon.a (mon.0) 491 : audit [DBG] from='client.? 172.21.15.29:0/2827676416' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T09:21:24.751 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:22.580608+0000 mgr.y (mgr.14144) 186 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:24.752 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:24.285003+0000 mon.a (mon.0) 491 : audit [DBG] from='client.? 172.21.15.29:0/2827676416' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T09:21:26.602 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:24.580907+0000 mgr.y (mgr.14144) 187 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:26.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:24.580907+0000 mgr.y (mgr.14144) 187 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:26.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:24.580907+0000 mgr.y (mgr.14144) 187 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:27.182 INFO:teuthology.orchestra.run.smithi029.stdout:206158430222 2024-09-17T09:21:27.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:27.180947+0000 mon.a (mon.0) 492 : audit [DBG] from='client.? 172.21.15.29:0/1154317075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T09:21:27.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:27.180947+0000 mon.a (mon.0) 492 : audit [DBG] from='client.? 172.21.15.29:0/1154317075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T09:21:27.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:27.180947+0000 mon.a (mon.0) 492 : audit [DBG] from='client.? 172.21.15.29:0/1154317075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T09:21:28.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:26.581272+0000 mgr.y (mgr.14144) 188 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:28.762 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:26.581272+0000 mgr.y (mgr.14144) 188 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:28.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:26.581272+0000 mgr.y (mgr.14144) 188 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:28.762 INFO:teuthology.orchestra.run.smithi029.stdout:107374182430 2024-09-17T09:21:29.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:28.581593+0000 mgr.y (mgr.14144) 189 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:29.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:28.761322+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.29:0/2317474416' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T09:21:29.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:28.581593+0000 mgr.y (mgr.14144) 189 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:29.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:28.761322+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.29:0/2317474416' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T09:21:29.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:28.581593+0000 mgr.y (mgr.14144) 189 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:29.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:28.761322+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.29:0/2317474416' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T09:21:30.902 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182427 got 107374182430 for osd.3 2024-09-17T09:21:30.902 DEBUG:teuthology.parallel:result is None 2024-09-17T09:21:31.293 INFO:teuthology.orchestra.run.smithi029.stdout:34359738408 2024-09-17T09:21:31.778 INFO:teuthology.orchestra.run.smithi029.stdout:77309411361 2024-09-17T09:21:32.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:30.582006+0000 mgr.y (mgr.14144) 190 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:32.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:31.292778+0000 mon.a (mon.0) 493 : audit [DBG] from='client.? 172.21.15.29:0/4271048029' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T09:21:32.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:30.582006+0000 mgr.y (mgr.14144) 190 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:32.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:31.292778+0000 mon.a (mon.0) 493 : audit [DBG] from='client.? 172.21.15.29:0/4271048029' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T09:21:32.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:30.582006+0000 mgr.y (mgr.14144) 190 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:32.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:31.292778+0000 mon.a (mon.0) 493 : audit [DBG] from='client.? 172.21.15.29:0/4271048029' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T09:21:34.271 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:32.582378+0000 mgr.y (mgr.14144) 191 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:34.271 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:32.582378+0000 mgr.y (mgr.14144) 191 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:34.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:32.582378+0000 mgr.y (mgr.14144) 191 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:35.149 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:34.547027+0000 mon.a (mon.0) 494 : audit [DBG] from='client.? 172.21.15.29:0/2090336415' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T09:21:35.211 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:34.547027+0000 mon.a (mon.0) 494 : audit [DBG] from='client.? 172.21.15.29:0/2090336415' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T09:21:35.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:34.547027+0000 mon.a (mon.0) 494 : audit [DBG] from='client.? 172.21.15.29:0/2090336415' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T09:21:35.506 INFO:teuthology.orchestra.run.smithi029.stdout:55834574887 2024-09-17T09:21:36.034 INFO:teuthology.orchestra.run.smithi029.stdout:180388626453 2024-09-17T09:21:36.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:34.582664+0000 mgr.y (mgr.14144) 192 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:34.582664+0000 mgr.y (mgr.14144) 192 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:36.399 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738404 got 34359738408 for osd.0 2024-09-17T09:21:36.399 DEBUG:teuthology.parallel:result is None 2024-09-17T09:21:36.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:34.582664+0000 mgr.y (mgr.14144) 192 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:37.032 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626448 got 180388626453 for osd.6 2024-09-17T09:21:37.033 DEBUG:teuthology.parallel:result is None 2024-09-17T09:21:37.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:36.034086+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.29:0/3799976872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T09:21:37.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:36.034086+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.29:0/3799976872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T09:21:37.450 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430219 got 206158430222 for osd.7 2024-09-17T09:21:37.450 DEBUG:teuthology.parallel:result is None 2024-09-17T09:21:37.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:36.034086+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.29:0/3799976872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T09:21:38.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:36.583033+0000 mgr.y (mgr.14144) 193 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:38.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:36.583033+0000 mgr.y (mgr.14144) 193 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:38.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:36.583033+0000 mgr.y (mgr.14144) 193 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:38.716 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574883 got 55834574887 for osd.1 2024-09-17T09:21:38.716 DEBUG:teuthology.parallel:result is None 2024-09-17T09:21:39.095 INFO:teuthology.orchestra.run.smithi029.stdout:154618822679 2024-09-17T09:21:40.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:38.583355+0000 mgr.y (mgr.14144) 194 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:40.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:39.095275+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.29:0/4050346216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T09:21:40.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:38.583355+0000 mgr.y (mgr.14144) 194 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:40.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:40 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:39.095275+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.29:0/4050346216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T09:21:40.446 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411359 got 77309411361 for osd.2 2024-09-17T09:21:40.447 DEBUG:teuthology.parallel:result is None 2024-09-17T09:21:40.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:38.583355+0000 mgr.y (mgr.14144) 194 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:40.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:39.095275+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.29:0/4050346216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T09:21:41.155 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822674 got 154618822679 for osd.5 2024-09-17T09:21:41.155 DEBUG:teuthology.parallel:result is None 2024-09-17T09:21:42.144 INFO:teuthology.orchestra.run.smithi029.stdout:128849018909 2024-09-17T09:21:42.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:40.583803+0000 mgr.y (mgr.14144) 195 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:42.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:40.583803+0000 mgr.y (mgr.14144) 195 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:42.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:42 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:40.583803+0000 mgr.y (mgr.14144) 195 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:42.802 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018904 got 128849018909 for osd.4 2024-09-17T09:21:42.802 DEBUG:teuthology.parallel:result is None 2024-09-17T09:21:42.803 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-17T09:21:42.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T09:21:43.025 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:43 smithi029 podman[41208]: 2024-09-17 09:21:43.02476643 +0000 UTC m=+0.613449105 container create d64bbebef82ef5b2b6c7c52f3dd1a8230650b61c600761dffe5b3e468ddea825 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:21:43.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:42.139378+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.29:0/942486332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T09:21:43.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:42.139378+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.29:0/942486332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T09:21:43.446 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:21:43.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:42.139378+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.29:0/942486332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T09:21:43.592 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:43 smithi029 podman[41208]: 2024-09-17 09:21:43.338870807 +0000 UTC m=+0.927553482 container init d64bbebef82ef5b2b6c7c52f3dd1a8230650b61c600761dffe5b3e468ddea825 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:21:43.592 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:43 smithi029 podman[41208]: 2024-09-17 09:21:43.342300493 +0000 UTC m=+0.930983169 container start d64bbebef82ef5b2b6c7c52f3dd1a8230650b61c600761dffe5b3e468ddea825 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:21:43.593 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[41267]: level=info ts=2024-09-17T09:21:43.362Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2024-09-17T09:21:43.593 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[41267]: level=info ts=2024-09-17T09:21:43.362Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2024-09-17T09:21:43.593 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[41267]: level=info ts=2024-09-17T09:21:43.363Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.29 port=9094 2024-09-17T09:21:43.593 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[41267]: level=info ts=2024-09-17T09:21:43.364Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T09:21:43.593 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[41267]: level=info ts=2024-09-17T09:21:43.383Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T09:21:43.593 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[41267]: level=info ts=2024-09-17T09:21:43.383Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T09:21:43.593 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[41267]: level=info ts=2024-09-17T09:21:43.387Z caller=main.go:497 msg=Listening address=:9093 2024-09-17T09:21:43.593 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:43 smithi029 bash[41208]: d64bbebef82ef5b2b6c7c52f3dd1a8230650b61c600761dffe5b3e468ddea825 2024-09-17T09:21:43.593 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:43 smithi029 systemd[1]: Started Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:21:44.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:42.584160+0000 mgr.y (mgr.14144) 196 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:44.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:43.504673+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:44.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:43.599018+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:44.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:43.603552+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:44.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:43.604046+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:21:44.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:43.609815+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:44.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:42.584160+0000 mgr.y (mgr.14144) 196 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:44.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:43.504673+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:44.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:43.599018+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:44.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:43.603552+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:44.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:43.604046+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:21:44.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:43.609815+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:44.407 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:42.584160+0000 mgr.y (mgr.14144) 196 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:44.407 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:43.504673+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:44.407 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:43.599018+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:44.407 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:43.603552+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:44.407 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:43.604046+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:21:44.407 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:43.609815+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:45.212 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:21:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:43.604519+0000 mgr.y (mgr.14144) 197 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:21:45.212 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:21:43.613406+0000 mgr.y (mgr.14144) 198 : cephadm [INF] Deploying daemon grafana.a on smithi103 2024-09-17T09:21:45.213 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:43.604519+0000 mgr.y (mgr.14144) 197 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:21:45.213 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:21:43.613406+0000 mgr.y (mgr.14144) 198 : cephadm [INF] Deploying daemon grafana.a on smithi103 2024-09-17T09:21:45.214 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-17T09:21:45.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:43.604519+0000 mgr.y (mgr.14144) 197 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:21:45.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:21:43.613406+0000 mgr.y (mgr.14144) 198 : cephadm [INF] Deploying daemon grafana.a on smithi103 2024-09-17T09:21:45.535 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[41267]: level=info ts=2024-09-17T09:21:45.364Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00009646s 2024-09-17T09:21:45.848 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-17T09:21:44.584340+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":43224,"kb_used_data":3736,"kb_used_omap":0,"kb_used_meta":39424,"kb_avail":749917992,"statfs":{"total":767960285184,"available":767916023808,"internally_reserved":0,"allocated":3825664,"data_stored":1511330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40370176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"50","state":"active+clean","last_fresh":"2024-09-17T09:20:28.423944+0000","last_change":"2024-09-17T09:20:28.423944+0000","last_active":"2024-09-17T09:20:28.423944+0000","last_peered":"2024-09-17T09:20:28.423944+0000","last_clean":"2024-09-17T09:20:28.423944+0000","last_became_active":"2024-09-17T09:20:28.423571+0000","last_became_peered":"2024-09-17T09:20:28.423571+0000","last_unstale":"2024-09-17T09:20:28.423944+0000","last_undegraded":"2024-09-17T09:20:28.423944+0000","last_fullsized":"2024-09-17T09:20:28.423944+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:18:56.086750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:18:56.086750+0000","last_clean_scrub_stamp":"2024-09-17T09:18:56.086750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5208,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93739944,"statfs":{"total":95995035648,"available":95989702656,"internally_reserved":0,"allocated":475136,"data_stored":188680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.55500000000000005}]},{"osd":1,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.50900000000000001}]},{"osd":2,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.44},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.67100000000000004}]},{"osd":3,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.496},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.64300000000000002}]},{"osd":4,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.61599999999999999}]},{"osd":5,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.56599999999999995}]}]},{"osd":6,"up_from":42,"seq":180388626455,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5272,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739880,"statfs":{"total":95995035648,"available":95989637120,"internally_reserved":0,"allocated":475136,"data_stored":188680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.39100000000000001}]},{"osd":1,"last update":"Tue Sep 17 09:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.442}]},{"osd":2,"last update":"Tue Sep 17 09:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.40500000000000003}]},{"osd":3,"last update":"Tue Sep 17 09:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.502}]},{"osd":4,"last update":"Tue Sep 17 09:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.58999999999999997}]},{"osd":5,"last update":"Tue Sep 17 09:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.64000000000000001}]},{"osd":7,"last update":"Tue Sep 17 09:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.57499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5532,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739620,"statfs":{"total":95995035648,"available":95989370880,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:20:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.34699999999999998,"15min":0.33700000000000002},"min":{"1min":0.314,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.53400000000000003,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.36599999999999999,"15min":0.35299999999999998},"min":{"1min":0.33100000000000002,"5min":0.217,"15min":0.217},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.39500000000000002}]},{"osd":2,"last update":"Tue Sep 17 09:21:02 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.47199999999999998,"15min":0.46700000000000003},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.441},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.253,"15min":0.253},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.45100000000000001}]},{"osd":3,"last update":"Tue Sep 17 09:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.46800000000000003,"15min":0.45400000000000001},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":2.2530000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.48299999999999998,"15min":0.47099999999999997},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.3159999999999998,"5min":2.3159999999999998,"15min":2.3159999999999998},"last":0.46999999999999997}]},{"osd":4,"last update":"Tue Sep 17 09:21:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.60499999999999998,"15min":0.59499999999999997},"min":{"1min":0.48799999999999999,"5min":0.435,"15min":0.435},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.61699999999999999}]},{"osd":5,"last update":"Tue Sep 17 09:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.68500000000000005}]},{"osd":6,"last update":"Tue Sep 17 09:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.626},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.54100000000000004}]},{"osd":7,"last update":"Tue Sep 17 09:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.56699999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738411,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5596,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739556,"statfs":{"total":95995035648,"available":95989305344,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 09:20:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.34499999999999997,"15min":0.32700000000000001},"min":{"1min":0.28000000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.443,"5min":0.35999999999999999,"15min":0.34599999999999997},"min":{"1min":0.33000000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.376}]},{"osd":2,"last update":"Tue Sep 17 09:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.41899999999999998},"min":{"1min":0.315,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.432},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.39400000000000002},"min":{"1min":0.28499999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.58899999999999997,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.41099999999999998}]},{"osd":3,"last update":"Tue Sep 17 09:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.436,"15min":0.441},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60899999999999999,"5min":0.628,"15min":0.628},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.32900000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.48799999999999999}]},{"osd":4,"last update":"Tue Sep 17 09:21:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56899999999999995,"15min":0.56399999999999995},"min":{"1min":0.46300000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.74299999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63300000000000001,"15min":0.63200000000000001},"min":{"1min":0.51100000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61699999999999999}]},{"osd":5,"last update":"Tue Sep 17 09:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.57399999999999995}]},{"osd":6,"last update":"Tue Sep 17 09:21:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.504},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.66100000000000003}]},{"osd":7,"last update":"Tue Sep 17 09:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.63100000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411365,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5468,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739684,"statfs":{"total":95995035648,"available":95989436416,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.38,"15min":0.371},"min":{"1min":0.29799999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.29799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.42199999999999999}]},{"osd":1,"last update":"Tue Sep 17 09:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.25700000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.436},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.40400000000000003,"15min":0.39500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.56000000000000005}]},{"osd":3,"last update":"Tue Sep 17 09:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.42999999999999999,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57099999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.36699999999999999}]},{"osd":4,"last update":"Tue Sep 17 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.57899999999999996,"15min":0.56200000000000006},"min":{"1min":0.47999999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.621,"15min":0.60799999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.59099999999999997}]},{"osd":5,"last update":"Tue Sep 17 09:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.625},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.57999999999999996}]},{"osd":6,"last update":"Tue Sep 17 09:21:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.69899999999999995}]},{"osd":7,"last update":"Tue Sep 17 09:21:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.63500000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182433,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5404,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739748,"statfs":{"total":95995035648,"available":95989501952,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.39800000000000002,"15min":0.39300000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.41799999999999998,"15min":0.42399999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.56399999999999995}]},{"osd":1,"last update":"Tue Sep 17 09:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40600000000000003,"15min":0.40899999999999997},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56499999999999995,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.433,"15min":0.434},"min":{"1min":0.27100000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.63500000000000001,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Tue Sep 17 09:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46600000000000003,"15min":0.46700000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.437,"5min":0.45900000000000002,"15min":0.46300000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64200000000000002,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.50600000000000001}]},{"osd":4,"last update":"Tue Sep 17 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.39200000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81799999999999995,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.41199999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.65500000000000003}]},{"osd":5,"last update":"Tue Sep 17 09:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.752}]},{"osd":6,"last update":"Tue Sep 17 09:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.72299999999999998}]},{"osd":7,"last update":"Tue Sep 17 09:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.70799999999999996}]}]},{"osd":4,"up_from":30,"seq":128849018909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5404,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739748,"statfs":{"total":95995035648,"available":95989501952,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:21:34 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.53700000000000003,"15min":0.54300000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68500000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53900000000000003,"15min":0.53500000000000003},"min":{"1min":0.39000000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72599999999999998,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.70299999999999996}]},{"osd":1,"last update":"Tue Sep 17 09:21:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53600000000000003,"15min":0.53700000000000003},"min":{"1min":0.374,"5min":0.37,"15min":0.37},"max":{"1min":0.70899999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55200000000000005,"15min":0.55400000000000005},"min":{"1min":0.39800000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72499999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.72899999999999998}]},{"osd":2,"last update":"Tue Sep 17 09:21:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.752,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.622},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54600000000000004,"15min":0.54900000000000004},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68400000000000005,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56200000000000006}]},{"osd":3,"last update":"Tue Sep 17 09:21:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54600000000000004,"15min":0.54800000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66800000000000004,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.40699999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.70099999999999996,"5min":0.871,"15min":0.871},"last":0.64700000000000002}]},{"osd":5,"last update":"Tue Sep 17 09:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.60799999999999998}]},{"osd":6,"last update":"Tue Sep 17 09:21:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.75}]},{"osd":7,"last update":"Tue Sep 17 09:21:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.71699999999999997}]}]},{"osd":5,"up_from":36,"seq":154618822680,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.70199999999999996}]},{"osd":1,"last update":"Tue Sep 17 09:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.67000000000000004}]},{"osd":2,"last update":"Tue Sep 17 09:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.626},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.55300000000000005}]},{"osd":3,"last update":"Tue Sep 17 09:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.68200000000000005}]},{"osd":4,"last update":"Tue Sep 17 09:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.35499999999999998}]},{"osd":6,"last update":"Tue Sep 17 09:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.46500000000000002}]},{"osd":7,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T09:21:45.850 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T09:21:46.062 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:21:46.183 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:44.584499+0000 mgr.y (mgr.14144) 199 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:46.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:45.208264+0000 mgr.y (mgr.14144) 200 : audit [DBG] from='client.14538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:21:46.184 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:44.584499+0000 mgr.y (mgr.14144) 199 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:46.184 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:45.208264+0000 mgr.y (mgr.14144) 200 : audit [DBG] from='client.14538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:21:46.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:44.584499+0000 mgr.y (mgr.14144) 199 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:46.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:45.208264+0000 mgr.y (mgr.14144) 200 : audit [DBG] from='client.14538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:21:47.976 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:21:47.977 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-17T09:21:48.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:46.585001+0000 mgr.y (mgr.14144) 201 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:48.277 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:46.585001+0000 mgr.y (mgr.14144) 201 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:48.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:46.585001+0000 mgr.y (mgr.14144) 201 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:48.698 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-09-17T09:21:46.584743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":43224,"kb_used_data":3736,"kb_used_omap":0,"kb_used_meta":39424,"kb_avail":749917992,"statfs":{"total":767960285184,"available":767916023808,"internally_reserved":0,"allocated":3825664,"data_stored":1511330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40370176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"50","state":"active+clean","last_fresh":"2024-09-17T09:20:28.423944+0000","last_change":"2024-09-17T09:20:28.423944+0000","last_active":"2024-09-17T09:20:28.423944+0000","last_peered":"2024-09-17T09:20:28.423944+0000","last_clean":"2024-09-17T09:20:28.423944+0000","last_became_active":"2024-09-17T09:20:28.423571+0000","last_became_peered":"2024-09-17T09:20:28.423571+0000","last_unstale":"2024-09-17T09:20:28.423944+0000","last_undegraded":"2024-09-17T09:20:28.423944+0000","last_fullsized":"2024-09-17T09:20:28.423944+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T09:18:56.086750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T09:18:56.086750+0000","last_clean_scrub_stamp":"2024-09-17T09:18:56.086750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5208,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93739944,"statfs":{"total":95995035648,"available":95989702656,"internally_reserved":0,"allocated":475136,"data_stored":188680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.55500000000000005}]},{"osd":1,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.50900000000000001}]},{"osd":2,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.44},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.67100000000000004}]},{"osd":3,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.496},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.64300000000000002}]},{"osd":4,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.61599999999999999}]},{"osd":5,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.56599999999999995}]}]},{"osd":6,"up_from":42,"seq":180388626455,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5272,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":93739880,"statfs":{"total":95995035648,"available":95989637120,"internally_reserved":0,"allocated":475136,"data_stored":188680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.39100000000000001}]},{"osd":1,"last update":"Tue Sep 17 09:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.442}]},{"osd":2,"last update":"Tue Sep 17 09:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.40500000000000003}]},{"osd":3,"last update":"Tue Sep 17 09:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.502}]},{"osd":4,"last update":"Tue Sep 17 09:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.58999999999999997}]},{"osd":5,"last update":"Tue Sep 17 09:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.64000000000000001}]},{"osd":7,"last update":"Tue Sep 17 09:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.57499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5532,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739620,"statfs":{"total":95995035648,"available":95989370880,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:21:41 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.35999999999999999,"15min":0.34200000000000003},"min":{"1min":0.25,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.40100000000000002,"15min":0.36499999999999999},"min":{"1min":0.308,"5min":0.217,"15min":0.217},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":0.29099999999999998}]},{"osd":2,"last update":"Tue Sep 17 09:21:02 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.47199999999999998,"15min":0.46700000000000003},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.41999999999999998,"15min":0.41299999999999998},"min":{"1min":0.307,"5min":0.253,"15min":0.253},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.317}]},{"osd":3,"last update":"Tue Sep 17 09:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.46800000000000003,"15min":0.45400000000000001},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":2.2530000000000001},"last":0.308},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.48299999999999998,"15min":0.47099999999999997},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.3159999999999998,"5min":2.3159999999999998,"15min":2.3159999999999998},"last":0.377}]},{"osd":4,"last update":"Tue Sep 17 09:21:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.60499999999999998,"15min":0.59499999999999997},"min":{"1min":0.48799999999999999,"5min":0.435,"15min":0.435},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.58199999999999996}]},{"osd":5,"last update":"Tue Sep 17 09:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.55900000000000005}]},{"osd":6,"last update":"Tue Sep 17 09:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.60299999999999998}]},{"osd":7,"last update":"Tue Sep 17 09:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.64300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738411,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5596,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739556,"statfs":{"total":95995035648,"available":95989305344,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 09:20:43 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.34499999999999997,"15min":0.32700000000000001},"min":{"1min":0.28000000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.443,"5min":0.35999999999999999,"15min":0.34599999999999997},"min":{"1min":0.33000000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.376}]},{"osd":2,"last update":"Tue Sep 17 09:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.42299999999999999,"15min":0.41899999999999998},"min":{"1min":0.315,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.432},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.39400000000000002},"min":{"1min":0.28499999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.58899999999999997,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.41099999999999998}]},{"osd":3,"last update":"Tue Sep 17 09:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.436,"15min":0.441},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60899999999999999,"5min":0.628,"15min":0.628},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.32900000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.48799999999999999}]},{"osd":4,"last update":"Tue Sep 17 09:21:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56899999999999995,"15min":0.56399999999999995},"min":{"1min":0.46300000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.74299999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63300000000000001,"15min":0.63200000000000001},"min":{"1min":0.51100000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61699999999999999}]},{"osd":5,"last update":"Tue Sep 17 09:20:52 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.57399999999999995}]},{"osd":6,"last update":"Tue Sep 17 09:21:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.504},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.66100000000000003}]},{"osd":7,"last update":"Tue Sep 17 09:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.63100000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411366,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5468,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739684,"statfs":{"total":95995035648,"available":95989436416,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.38,"15min":0.371},"min":{"1min":0.29799999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.29799999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.40799999999999997}]},{"osd":1,"last update":"Tue Sep 17 09:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.25700000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.40400000000000003,"15min":0.39500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.29699999999999999}]},{"osd":3,"last update":"Tue Sep 17 09:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.42999999999999999,"15min":0.42799999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57099999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.28199999999999997}]},{"osd":4,"last update":"Tue Sep 17 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.57899999999999996,"15min":0.56200000000000006},"min":{"1min":0.47999999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.621,"15min":0.60799999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.45300000000000001}]},{"osd":5,"last update":"Tue Sep 17 09:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.438}]},{"osd":6,"last update":"Tue Sep 17 09:21:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.54000000000000004}]},{"osd":7,"last update":"Tue Sep 17 09:21:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.55100000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182434,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5404,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739748,"statfs":{"total":95995035648,"available":95989501952,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.39800000000000002,"15min":0.39300000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.41799999999999998,"15min":0.42399999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.35899999999999999}]},{"osd":1,"last update":"Tue Sep 17 09:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.40600000000000003,"15min":0.40899999999999997},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.56499999999999995,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.433,"15min":0.434},"min":{"1min":0.27100000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.63500000000000001,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.317}]},{"osd":2,"last update":"Tue Sep 17 09:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46600000000000003,"15min":0.46700000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.45900000000000002,"15min":0.46300000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64200000000000002,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.308}]},{"osd":4,"last update":"Tue Sep 17 09:21:37 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.39200000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81799999999999995,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59599999999999997,"15min":0.59399999999999997},"min":{"1min":0.41199999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.41799999999999998}]},{"osd":5,"last update":"Tue Sep 17 09:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.48999999999999999}]},{"osd":6,"last update":"Tue Sep 17 09:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.57199999999999995}]},{"osd":7,"last update":"Tue Sep 17 09:21:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.45800000000000002}]}]},{"osd":4,"up_from":30,"seq":128849018910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5404,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739748,"statfs":{"total":95995035648,"available":95989501952,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:21:34 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.53700000000000003,"15min":0.54300000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68500000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53900000000000003,"15min":0.53500000000000003},"min":{"1min":0.39000000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72599999999999998,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.53600000000000003}]},{"osd":1,"last update":"Tue Sep 17 09:21:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53600000000000003,"15min":0.53700000000000003},"min":{"1min":0.374,"5min":0.37,"15min":0.37},"max":{"1min":0.70899999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55200000000000005,"15min":0.55400000000000005},"min":{"1min":0.39800000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72499999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.50900000000000001}]},{"osd":2,"last update":"Tue Sep 17 09:21:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.752,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54600000000000004,"15min":0.54900000000000004},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68400000000000005,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.52600000000000002}]},{"osd":3,"last update":"Tue Sep 17 09:21:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54600000000000004,"15min":0.54800000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66800000000000004,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.40699999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.70099999999999996,"5min":0.871,"15min":0.871},"last":0.56499999999999995}]},{"osd":5,"last update":"Tue Sep 17 09:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.307}]},{"osd":6,"last update":"Tue Sep 17 09:21:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.34899999999999998}]},{"osd":7,"last update":"Tue Sep 17 09:21:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.38},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.371}]}]},{"osd":5,"up_from":36,"seq":154618822681,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":479232,"data_stored":188995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 09:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.56699999999999995}]},{"osd":1,"last update":"Tue Sep 17 09:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.53800000000000003}]},{"osd":2,"last update":"Tue Sep 17 09:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51500000000000001}]},{"osd":3,"last update":"Tue Sep 17 09:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.503},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.54400000000000004}]},{"osd":4,"last update":"Tue Sep 17 09:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.371}]},{"osd":6,"last update":"Tue Sep 17 09:21:12 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.41099999999999998}]},{"osd":7,"last update":"Tue Sep 17 09:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.39900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T09:21:48.699 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-17T09:21:48.700 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T09:21:48.700 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-17T09:21:48.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-17T09:21:48.907 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T09:21:49.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:47.971940+0000 mgr.y (mgr.14144) 202 : audit [DBG] from='client.24463 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:21:49.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:47.971940+0000 mgr.y (mgr.14144) 202 : audit [DBG] from='client.24463 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:21:49.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:47.971940+0000 mgr.y (mgr.14144) 202 : audit [DBG] from='client.24463 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T09:21:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:48.585318+0000 mgr.y (mgr.14144) 203 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:50.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:48.585318+0000 mgr.y (mgr.14144) 203 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:50.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:48.585318+0000 mgr.y (mgr.14144) 203 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:50.759 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-17T09:21:51.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:50.758250+0000 mon.a (mon.0) 500 : audit [DBG] from='client.? 172.21.15.29:0/1024961258' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T09:21:51.428 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:50.758250+0000 mon.a (mon.0) 500 : audit [DBG] from='client.? 172.21.15.29:0/1024961258' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T09:21:51.455 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T09:21:51.455 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-17T09:21:51.455 INFO:tasks.cephadm:Setup complete, yielding 2024-09-17T09:21:51.456 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T09:21:51.468 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-09-17T09:21:51.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-09-17T09:21:51.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:50.758250+0000 mon.a (mon.0) 500 : audit [DBG] from='client.? 172.21.15.29:0/1024961258' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T09:21:52.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:50.585575+0000 mgr.y (mgr.14144) 204 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:52.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:50.585575+0000 mgr.y (mgr.14144) 204 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:52.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:50.585575+0000 mgr.y (mgr.14144) 204 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:53.775 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:21:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[41267]: level=info ts=2024-09-17T09:21:53.365Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000708911s 2024-09-17T09:21:54.197 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T09:21:54.207 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-09-17T09:21:54.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-09-17T09:21:54.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:52.585965+0000 mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:54.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:53.464941+0000 mon.a (mon.0) 501 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T09:21:54.513 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:52.585965+0000 mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:54.513 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:53.464941+0000 mon.a (mon.0) 501 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T09:21:54.513 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:52.585965+0000 mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:54.513 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:53.464941+0000 mon.a (mon.0) 501 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T09:21:56.152 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:54.586237+0000 mgr.y (mgr.14144) 206 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:56.152 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:54.586237+0000 mgr.y (mgr.14144) 206 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:56.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:54.586237+0000 mgr.y (mgr.14144) 206 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:58.103 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:21:58.103 INFO:teuthology.orchestra.run.smithi029.stdout: "id": "d084521a-8d82-4dc2-bdd6-3f665e8bd2be", 2024-09-17T09:21:58.103 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "r", 2024-09-17T09:21:58.103 INFO:teuthology.orchestra.run.smithi029.stdout: "current_period": "8aac02fd-4bc3-484e-b246-d0c65dbac374", 2024-09-17T09:21:58.103 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 1 2024-09-17T09:21:58.103 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:21:58.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:56.586488+0000 mgr.y (mgr.14144) 207 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:58.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:57.065258+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2024-09-17T09:21:58.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:57.066974+0000 mon.a (mon.0) 503 : audit [INF] from='client.? 172.21.15.29:0/1906944054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T09:21:58.391 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:56.586488+0000 mgr.y (mgr.14144) 207 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:58.391 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:57.065258+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2024-09-17T09:21:58.391 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:57.066974+0000 mon.a (mon.0) 503 : audit [INF] from='client.? 172.21.15.29:0/1906944054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T09:21:58.391 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:56.586488+0000 mgr.y (mgr.14144) 207 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:21:58.391 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:57.065258+0000 mon.a (mon.0) 502 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2024-09-17T09:21:58.391 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:57.066974+0000 mon.a (mon.0) 503 : audit [INF] from='client.? 172.21.15.29:0/1906944054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T09:21:58.591 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 podman[36509]: 2024-09-17 09:21:58.481508039 +0000 UTC m=+0.141786911 container create b74e78c8a6b5eee4540b90dafd8f6a324f70299107877e8d6c16637307145930 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.license=GPLv2, description=Ceph Grafana Container, org.label-schema.build-date=20201204, io.buildah.version=1.19.8, maintainer=Paul Cuzner , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image) 2024-09-17T09:21:58.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-09-17T09:21:58.841 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 podman[36509]: 2024-09-17 09:21:58.701892787 +0000 UTC m=+0.362171664 container init b74e78c8a6b5eee4540b90dafd8f6a324f70299107877e8d6c16637307145930 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a, org.label-schema.build-date=20201204, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.license=GPLv2, maintainer=Paul Cuzner , description=Ceph Grafana Container, io.buildah.version=1.19.8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image) 2024-09-17T09:21:58.841 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 podman[36509]: 2024-09-17 09:21:58.704704574 +0000 UTC m=+0.364983447 container start b74e78c8a6b5eee4540b90dafd8f6a324f70299107877e8d6c16637307145930 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, io.buildah.version=1.19.8, maintainer=Paul Cuzner , org.label-schema.license=GPLv2, description=Ceph Grafana Container, org.label-schema.build-date=20201204, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="App mode production" logger=settings 2024-09-17T09:21:58.842 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing SqlStore" logger=server 2024-09-17T09:21:58.844 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-09-17T09:21:58.844 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Starting DB migration" logger=migrator 2024-09-17T09:21:58.844 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-09-17T09:21:58.844 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-09-17T09:21:58.844 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-09-17T09:21:58.844 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-09-17T09:21:58.844 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-09-17T09:21:58.844 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-09-17T09:21:58.845 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-09-17T09:21:58.845 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-09-17T09:21:58.845 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-09-17T09:21:58.845 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-09-17T09:21:58.845 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-09-17T09:21:58.845 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-09-17T09:21:58.845 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-09-17T09:21:58.845 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-09-17T09:21:58.845 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-09-17T09:21:58.845 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-09-17T09:21:58.845 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-09-17T09:21:58.845 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-09-17T09:21:58.846 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-09-17T09:21:58.847 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-09-17T09:21:58.847 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-09-17T09:21:58.847 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-09-17T09:21:58.847 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-09-17T09:21:58.847 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-09-17T09:21:58.847 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-09-17T09:21:58.847 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-17T09:21:58.847 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-09-17T09:21:58.847 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-09-17T09:21:58.847 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-09-17T09:21:58.847 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-09-17T09:21:58.847 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-09-17T09:21:58.847 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-09-17T09:21:58.848 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-09-17T09:21:58.848 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-09-17T09:21:58.848 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-09-17T09:21:58.848 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-09-17T09:21:58.848 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-09-17T09:21:58.848 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-09-17T09:21:58.848 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-09-17T09:21:58.848 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-09-17T09:21:58.848 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-09-17T09:21:58.848 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-09-17T09:21:58.848 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-09-17T09:21:58.849 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-09-17T09:21:58.849 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-09-17T09:21:58.849 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-09-17T09:21:58.850 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-09-17T09:21:58.850 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-09-17T09:21:58.850 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-09-17T09:21:58.850 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-09-17T09:21:58.851 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-17T09:21:58.851 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-09-17T09:21:58.851 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-17T09:21:58.851 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-09-17T09:21:58.851 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-17T09:21:58.851 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-17T09:21:58.851 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-09-17T09:21:58.851 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-09-17T09:21:58.851 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-09-17T09:21:58.851 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-09-17T09:21:58.851 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-09-17T09:21:58.851 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-09-17T09:21:58.851 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-09-17T09:21:58.852 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-09-17T09:21:58.852 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-09-17T09:21:58.852 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-09-17T09:21:58.852 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-09-17T09:21:58.852 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-09-17T09:21:58.852 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-09-17T09:21:58.852 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-09-17T09:21:58.852 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-09-17T09:21:58.852 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-09-17T09:21:58.852 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-09-17T09:21:58.852 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-09-17T09:21:58.852 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-09-17T09:21:58.852 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-09-17T09:21:58.853 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-09-17T09:21:58.853 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-09-17T09:21:58.853 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-09-17T09:21:58.853 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-09-17T09:21:58.853 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-09-17T09:21:58.853 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-09-17T09:21:58.853 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-09-17T09:21:58.853 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-09-17T09:21:58.853 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-09-17T09:21:58.853 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-09-17T09:21:58.853 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-09-17T09:21:58.853 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-09-17T09:21:58.854 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-09-17T09:21:58.854 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-09-17T09:21:58.854 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-09-17T09:21:58.854 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-09-17T09:21:58.854 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-09-17T09:21:58.854 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-09-17T09:21:58.854 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-09-17T09:21:58.854 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-09-17T09:21:58.854 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-09-17T09:21:58.854 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-17T09:21:58.854 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-17T09:21:58.854 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-09-17T09:21:58.854 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-09-17T09:21:58.855 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-09-17T09:21:58.856 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-09-17T09:21:58.856 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-09-17T09:21:58.856 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-09-17T09:21:58.856 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-09-17T09:21:58.856 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-09-17T09:21:58.856 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-09-17T09:21:58.856 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-09-17T09:21:58.856 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-09-17T09:21:58.856 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-09-17T09:21:58.856 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-09-17T09:21:58.856 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-09-17T09:21:58.856 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-09-17T09:21:58.857 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-17T09:21:58.857 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-09-17T09:21:58.857 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-09-17T09:21:58.857 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-09-17T09:21:58.857 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-09-17T09:21:58.857 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-09-17T09:21:58.857 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-09-17T09:21:58.857 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-09-17T09:21:58.857 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-09-17T09:21:58.857 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-09-17T09:21:58.857 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-17T09:21:58.857 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-09-17T09:21:58.857 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-09-17T09:21:58.858 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-09-17T09:21:58.859 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-09-17T09:21:58.860 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 bash[36509]: b74e78c8a6b5eee4540b90dafd8f6a324f70299107877e8d6c16637307145930 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 systemd[1]: Started Ceph grafana.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-17T09:21:58.861 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-09-17T09:21:58.862 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-09-17T09:21:58.862 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-09-17T09:21:58.862 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-09-17T09:21:58.862 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-09-17T09:21:58.862 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-09-17T09:21:58.862 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-17T09:21:58.862 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-09-17T09:21:58.862 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-09-17T09:21:58.862 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-09-17T09:21:58.862 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-09-17T09:21:58.862 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-09-17T09:21:58.862 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-17T09:21:58.862 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-09-17T09:21:58.863 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-09-17T09:21:58.863 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-09-17T09:21:58.863 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-09-17T09:21:58.863 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-09-17T09:21:58.863 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-09-17T09:21:58.863 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-09-17T09:21:58.863 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-09-17T09:21:58.863 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-09-17T09:21:58.863 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-09-17T09:21:58.863 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-09-17T09:21:58.863 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-09-17T09:21:58.863 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-09-17T09:21:58.863 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Created default admin" logger=sqlstore user=admin 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing HTTPServer" logger=server 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing PluginManager" logger=server 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Starting plugin search" logger=plugins 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing HooksService" logger=server 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing RemoteCache" logger=server 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing RenderingService" logger=server 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing AlertEngine" logger=server 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing QuotaService" logger=server 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing ServerLockService" logger=server 2024-09-17T09:21:58.864 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2024-09-17T09:21:58.865 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2024-09-17T09:21:58.865 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing LoginService" logger=server 2024-09-17T09:21:58.865 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing SearchService" logger=server 2024-09-17T09:21:58.865 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing TracingService" logger=server 2024-09-17T09:21:58.865 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2024-09-17T09:21:58.865 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing CleanUpService" logger=server 2024-09-17T09:21:58.865 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing NotificationService" logger=server 2024-09-17T09:21:58.865 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2024-09-17T09:21:58.865 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-17T09:21:58.865 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2024-09-17T09:21:58.865 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2024-09-17T09:21:58.865 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2024-09-17T09:21:58.866 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="Initializing Stream Manager" 2024-09-17T09:21:58.866 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:21:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:21:58+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-09-17T09:21:59.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:58.067835+0000 mon.a (mon.0) 504 : audit [INF] from='client.? 172.21.15.29:0/1906944054' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T09:21:59.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:58.068565+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-09-17T09:21:59.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:21:58.858757+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:59.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:21:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:59.067971+0000 mon.a (mon.0) 507 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-09-17T09:21:59.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:58.067835+0000 mon.a (mon.0) 504 : audit [INF] from='client.? 172.21.15.29:0/1906944054' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T09:21:59.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:58.068565+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-09-17T09:21:59.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:21:58.858757+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:59.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:21:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:59.067971+0000 mon.a (mon.0) 507 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-09-17T09:21:59.350 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:58.067835+0000 mon.a (mon.0) 504 : audit [INF] from='client.? 172.21.15.29:0/1906944054' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T09:21:59.350 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:58.068565+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-09-17T09:21:59.351 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:21:58.858757+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:21:59.351 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:21:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:59.067971+0000 mon.a (mon.0) 507 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-09-17T09:22:00.193 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:21:58.586910+0000 mgr.y (mgr.14144) 208 : cluster [DBG] pgmap v189: 33 pgs: 6 creating+peering, 21 unknown, 6 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:22:00.193 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:21:58.586910+0000 mgr.y (mgr.14144) 208 : cluster [DBG] pgmap v189: 33 pgs: 6 creating+peering, 21 unknown, 6 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:22:00.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:21:58.586910+0000 mgr.y (mgr.14144) 208 : cluster [DBG] pgmap v189: 33 pgs: 6 creating+peering, 21 unknown, 6 active+clean; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:22:00.685 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:22:00.685 INFO:teuthology.orchestra.run.smithi029.stdout: "id": "bd3db493-dd05-4278-aa3c-59043c019dcb", 2024-09-17T09:22:00.685 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default", 2024-09-17T09:22:00.685 INFO:teuthology.orchestra.run.smithi029.stdout: "api_name": "default", 2024-09-17T09:22:00.685 INFO:teuthology.orchestra.run.smithi029.stdout: "is_master": "true", 2024-09-17T09:22:00.685 INFO:teuthology.orchestra.run.smithi029.stdout: "endpoints": [], 2024-09-17T09:22:00.685 INFO:teuthology.orchestra.run.smithi029.stdout: "hostnames": [], 2024-09-17T09:22:00.685 INFO:teuthology.orchestra.run.smithi029.stdout: "hostnames_s3website": [], 2024-09-17T09:22:00.685 INFO:teuthology.orchestra.run.smithi029.stdout: "master_zone": "", 2024-09-17T09:22:00.685 INFO:teuthology.orchestra.run.smithi029.stdout: "zones": [], 2024-09-17T09:22:00.685 INFO:teuthology.orchestra.run.smithi029.stdout: "placement_targets": [], 2024-09-17T09:22:00.686 INFO:teuthology.orchestra.run.smithi029.stdout: "default_placement": "", 2024-09-17T09:22:00.686 INFO:teuthology.orchestra.run.smithi029.stdout: "realm_id": "d084521a-8d82-4dc2-bdd6-3f665e8bd2be", 2024-09-17T09:22:00.686 INFO:teuthology.orchestra.run.smithi029.stdout: "sync_policy": { 2024-09-17T09:22:00.686 INFO:teuthology.orchestra.run.smithi029.stdout: "groups": [] 2024-09-17T09:22:00.686 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:22:00.686 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:22:01.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-09-17T09:22:02.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:00.587212+0000 mgr.y (mgr.14144) 209 : cluster [DBG] pgmap v191: 33 pgs: 6 creating+peering, 12 unknown, 15 active+clean; 46 B data, 43 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:22:02.488 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:00.587212+0000 mgr.y (mgr.14144) 209 : cluster [DBG] pgmap v191: 33 pgs: 6 creating+peering, 12 unknown, 15 active+clean; 46 B data, 43 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:22:02.488 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:00.587212+0000 mgr.y (mgr.14144) 209 : cluster [DBG] pgmap v191: 33 pgs: 6 creating+peering, 12 unknown, 15 active+clean; 46 B data, 43 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:22:03.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:02.198473+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:03.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:02.198473+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:02.198473+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:03.951 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:22:03.951 INFO:teuthology.orchestra.run.smithi029.stdout: "id": "9380a79a-b931-4490-9bf2-22379873f915", 2024-09-17T09:22:03.951 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "z", 2024-09-17T09:22:03.951 INFO:teuthology.orchestra.run.smithi029.stdout: "domain_root": "z.rgw.meta:root", 2024-09-17T09:22:03.951 INFO:teuthology.orchestra.run.smithi029.stdout: "control_pool": "z.rgw.control", 2024-09-17T09:22:03.951 INFO:teuthology.orchestra.run.smithi029.stdout: "gc_pool": "z.rgw.log:gc", 2024-09-17T09:22:03.952 INFO:teuthology.orchestra.run.smithi029.stdout: "lc_pool": "z.rgw.log:lc", 2024-09-17T09:22:03.952 INFO:teuthology.orchestra.run.smithi029.stdout: "log_pool": "z.rgw.log", 2024-09-17T09:22:03.952 INFO:teuthology.orchestra.run.smithi029.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-09-17T09:22:03.952 INFO:teuthology.orchestra.run.smithi029.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-09-17T09:22:03.952 INFO:teuthology.orchestra.run.smithi029.stdout: "roles_pool": "z.rgw.meta:roles", 2024-09-17T09:22:03.952 INFO:teuthology.orchestra.run.smithi029.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-09-17T09:22:03.952 INFO:teuthology.orchestra.run.smithi029.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-09-17T09:22:03.952 INFO:teuthology.orchestra.run.smithi029.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-09-17T09:22:03.952 INFO:teuthology.orchestra.run.smithi029.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-09-17T09:22:03.952 INFO:teuthology.orchestra.run.smithi029.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-09-17T09:22:03.952 INFO:teuthology.orchestra.run.smithi029.stdout: "otp_pool": "z.rgw.otp", 2024-09-17T09:22:03.952 INFO:teuthology.orchestra.run.smithi029.stdout: "system_key": { 2024-09-17T09:22:03.953 INFO:teuthology.orchestra.run.smithi029.stdout: "access_key": "", 2024-09-17T09:22:03.953 INFO:teuthology.orchestra.run.smithi029.stdout: "secret_key": "" 2024-09-17T09:22:03.953 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:22:03.953 INFO:teuthology.orchestra.run.smithi029.stdout: "placement_pools": [ 2024-09-17T09:22:03.953 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-17T09:22:03.953 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "default-placement", 2024-09-17T09:22:03.953 INFO:teuthology.orchestra.run.smithi029.stdout: "val": { 2024-09-17T09:22:03.953 INFO:teuthology.orchestra.run.smithi029.stdout: "index_pool": "z.rgw.buckets.index", 2024-09-17T09:22:03.953 INFO:teuthology.orchestra.run.smithi029.stdout: "storage_classes": { 2024-09-17T09:22:03.953 INFO:teuthology.orchestra.run.smithi029.stdout: "STANDARD": { 2024-09-17T09:22:03.953 INFO:teuthology.orchestra.run.smithi029.stdout: "data_pool": "z.rgw.buckets.data" 2024-09-17T09:22:03.953 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:22:03.953 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:22:03.954 INFO:teuthology.orchestra.run.smithi029.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-09-17T09:22:03.954 INFO:teuthology.orchestra.run.smithi029.stdout: "index_type": 0 2024-09-17T09:22:03.954 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:22:03.954 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:22:03.954 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:22:03.954 INFO:teuthology.orchestra.run.smithi029.stdout: "realm_id": "d084521a-8d82-4dc2-bdd6-3f665e8bd2be", 2024-09-17T09:22:03.954 INFO:teuthology.orchestra.run.smithi029.stdout: "notif_pool": "z.rgw.log:notif" 2024-09-17T09:22:03.954 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:22:04.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:02.587759+0000 mgr.y (mgr.14144) 210 : cluster [DBG] pgmap v192: 33 pgs: 33 active+clean; 562 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 3.5 KiB/s wr, 4 op/s 2024-09-17T09:22:04.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:02.587759+0000 mgr.y (mgr.14144) 210 : cluster [DBG] pgmap v192: 33 pgs: 33 active+clean; 562 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 3.5 KiB/s wr, 4 op/s 2024-09-17T09:22:04.543 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:02.587759+0000 mgr.y (mgr.14144) 210 : cluster [DBG] pgmap v192: 33 pgs: 33 active+clean; 562 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 3.5 KiB/s wr, 4 op/s 2024-09-17T09:22:04.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-09-17T09:22:05.455 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:04.547868+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-09-17T09:22:05.455 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:04.672707+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:22:05.455 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:04.673502+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:22:05.455 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:04.547868+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-09-17T09:22:05.455 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:04.672707+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:22:05.456 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:04.673502+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:22:05.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:04.547868+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-09-17T09:22:05.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:04.672707+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:22:05.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:04.673502+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:22:06.486 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:04.588129+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v193: 33 pgs: 33 active+clean; 726 B data, 43 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 2.0 KiB/s wr, 2 op/s 2024-09-17T09:22:06.486 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:05.209501+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-09-17T09:22:06.486 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:05.209596+0000 mon.a (mon.0) 513 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-09-17T09:22:06.486 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:04.588129+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v193: 33 pgs: 33 active+clean; 726 B data, 43 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 2.0 KiB/s wr, 2 op/s 2024-09-17T09:22:06.486 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:05.209501+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-09-17T09:22:06.486 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:05.209596+0000 mon.a (mon.0) 513 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-09-17T09:22:06.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:04.588129+0000 mgr.y (mgr.14144) 211 : cluster [DBG] pgmap v193: 33 pgs: 33 active+clean; 726 B data, 43 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 2.0 KiB/s wr, 2 op/s 2024-09-17T09:22:06.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:05.209501+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-09-17T09:22:06.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:05.209596+0000 mon.a (mon.0) 513 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-09-17T09:22:07.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:06.211836+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-09-17T09:22:07.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:06.211836+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-09-17T09:22:07.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:06.211836+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-09-17T09:22:08.635 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:06.588538+0000 mgr.y (mgr.14144) 212 : cluster [DBG] pgmap v196: 33 pgs: 1 peering, 32 active+clean; 864 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 2.0 KiB/s wr, 2 op/s 2024-09-17T09:22:08.635 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:07.423999+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-09-17T09:22:08.635 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:07.426550+0000 mon.a (mon.0) 516 : audit [INF] from='client.? 172.21.15.29:0/4077253528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T09:22:08.635 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:07.613921+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:08.635 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:07.834306+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:08.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:06.588538+0000 mgr.y (mgr.14144) 212 : cluster [DBG] pgmap v196: 33 pgs: 1 peering, 32 active+clean; 864 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 2.0 KiB/s wr, 2 op/s 2024-09-17T09:22:08.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:07.423999+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-09-17T09:22:08.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:07.426550+0000 mon.a (mon.0) 516 : audit [INF] from='client.? 172.21.15.29:0/4077253528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T09:22:08.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:07.613921+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:08.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:07.834306+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:08.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:06.588538+0000 mgr.y (mgr.14144) 212 : cluster [DBG] pgmap v196: 33 pgs: 1 peering, 32 active+clean; 864 B data, 43 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 2.0 KiB/s wr, 2 op/s 2024-09-17T09:22:08.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:07.423999+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-09-17T09:22:08.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:07.426550+0000 mon.a (mon.0) 516 : audit [INF] from='client.? 172.21.15.29:0/4077253528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T09:22:08.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:07.613921+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:08.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:07.834306+0000 mon.a (mon.0) 518 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:09.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:07.836168+0000 mgr.y (mgr.14144) 213 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T09:22:09.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:07.838475+0000 mgr.y (mgr.14144) 214 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi029 2024-09-17T09:22:09.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:08.426003+0000 mon.a (mon.0) 519 : audit [INF] from='client.? 172.21.15.29:0/4077253528' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-17T09:22:09.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:08.426156+0000 mon.a (mon.0) 520 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-09-17T09:22:09.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:07.836168+0000 mgr.y (mgr.14144) 213 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T09:22:09.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:07.838475+0000 mgr.y (mgr.14144) 214 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi029 2024-09-17T09:22:09.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:08.426003+0000 mon.a (mon.0) 519 : audit [INF] from='client.? 172.21.15.29:0/4077253528' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-17T09:22:09.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:08.426156+0000 mon.a (mon.0) 520 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-09-17T09:22:09.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:07.836168+0000 mgr.y (mgr.14144) 213 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T09:22:09.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:07.838475+0000 mgr.y (mgr.14144) 214 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi029 2024-09-17T09:22:09.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:08.426003+0000 mon.a (mon.0) 519 : audit [INF] from='client.? 172.21.15.29:0/4077253528' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-17T09:22:09.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:08.426156+0000 mon.a (mon.0) 520 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-09-17T09:22:10.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:08.588866+0000 mgr.y (mgr.14144) 215 : cluster [DBG] pgmap v199: 65 pgs: 30 unknown, 1 peering, 34 active+clean; 1.8 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-17T09:22:10.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:09.429416+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-09-17T09:22:10.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:09.431350+0000 mon.a (mon.0) 522 : audit [INF] from='client.? 172.21.15.29:0/4077253528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T09:22:10.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:08.588866+0000 mgr.y (mgr.14144) 215 : cluster [DBG] pgmap v199: 65 pgs: 30 unknown, 1 peering, 34 active+clean; 1.8 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-17T09:22:10.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:09.429416+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-09-17T09:22:10.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:09.431350+0000 mon.a (mon.0) 522 : audit [INF] from='client.? 172.21.15.29:0/4077253528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T09:22:10.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:08.588866+0000 mgr.y (mgr.14144) 215 : cluster [DBG] pgmap v199: 65 pgs: 30 unknown, 1 peering, 34 active+clean; 1.8 KiB data, 43 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-17T09:22:10.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:09.429416+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-09-17T09:22:10.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:09.431350+0000 mon.a (mon.0) 522 : audit [INF] from='client.? 172.21.15.29:0/4077253528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T09:22:11.147 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:11 smithi029 systemd[1]: Stopping Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:22:11.417 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[41267]: level=info ts=2024-09-17T09:22:11.146Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2024-09-17T09:22:11.417 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:11 smithi029 podman[42900]: 2024-09-17 09:22:11.177640916 +0000 UTC m=+0.153087305 container died d64bbebef82ef5b2b6c7c52f3dd1a8230650b61c600761dffe5b3e468ddea825 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:22:11.762 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:10.431808+0000 mon.a (mon.0) 523 : audit [INF] from='client.? 172.21.15.29:0/4077253528' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-17T09:22:11.762 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:10.431961+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-09-17T09:22:11.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:10.431808+0000 mon.a (mon.0) 523 : audit [INF] from='client.? 172.21.15.29:0/4077253528' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-17T09:22:11.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:10.431961+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-09-17T09:22:11.762 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:11 smithi029 podman[42900]: 2024-09-17 09:22:11.51505045 +0000 UTC m=+0.490496835 container cleanup d64bbebef82ef5b2b6c7c52f3dd1a8230650b61c600761dffe5b3e468ddea825 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:22:11.763 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:11 smithi029 podman[42900]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a 2024-09-17T09:22:12.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:10.431808+0000 mon.a (mon.0) 523 : audit [INF] from='client.? 172.21.15.29:0/4077253528' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-17T09:22:12.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:10.431961+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-09-17T09:22:12.023 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:11 smithi029 podman[42923]: 2024-09-17 09:22:11.761093442 +0000 UTC m=+0.583685661 container remove d64bbebef82ef5b2b6c7c52f3dd1a8230650b61c600761dffe5b3e468ddea825 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:22:12.023 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:11 smithi029 podman[42923]: 2024-09-17 09:22:11.862691229 +0000 UTC m=+0.685283413 volume remove d759ce5788972ae078fe2bb720139b3e7476c2ada0f21f48ce66c8b6c299fb9e 2024-09-17T09:22:12.023 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:11 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T09:22:12.023 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:11 smithi029 systemd[1]: Stopped Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:22:12.023 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:11 smithi029 systemd[1]: Starting Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:22:12.293 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 podman[42971]: 2024-09-17 09:22:12.022951039 +0000 UTC m=+0.096150823 volume create 010fb0c79b3fb2997a1908a686a58e8c9beb8fbf278b5a0bd1cb7a3e07b801a8 2024-09-17T09:22:12.293 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 podman[42971]: 2024-09-17 09:22:11.942978217 +0000 UTC m=+0.016178002 image pull 0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0 docker.io/prom/alertmanager:v0.20.0 2024-09-17T09:22:12.293 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 podman[42971]: 2024-09-17 09:22:12.107043441 +0000 UTC m=+0.180243225 container create a5a9a8d22f29626ff43a834df8d8be84037bb2a0f701de2c4968bf95ddaeabd7 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:22:12.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:10.589260+0000 mgr.y (mgr.14144) 216 : cluster [DBG] pgmap v202: 97 pgs: 4 creating+activating, 11 creating+peering, 29 unknown, 1 peering, 52 active+clean; 1.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-09-17T09:22:12.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:11.542969+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-09-17T09:22:12.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:11.546251+0000 mon.a (mon.0) 526 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:22:12.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:12.523758+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:12.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:10.589260+0000 mgr.y (mgr.14144) 216 : cluster [DBG] pgmap v202: 97 pgs: 4 creating+activating, 11 creating+peering, 29 unknown, 1 peering, 52 active+clean; 1.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-09-17T09:22:12.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:11.542969+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-09-17T09:22:12.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:11.546251+0000 mon.a (mon.0) 526 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:22:12.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:12.523758+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:12.549 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 podman[42971]: 2024-09-17 09:22:12.396041026 +0000 UTC m=+0.469240811 container init a5a9a8d22f29626ff43a834df8d8be84037bb2a0f701de2c4968bf95ddaeabd7 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:22:12.549 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 podman[42971]: 2024-09-17 09:22:12.399078765 +0000 UTC m=+0.472278552 container start a5a9a8d22f29626ff43a834df8d8be84037bb2a0f701de2c4968bf95ddaeabd7 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:22:12.549 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[42985]: level=info ts=2024-09-17T09:22:12.419Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2024-09-17T09:22:12.549 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[42985]: level=info ts=2024-09-17T09:22:12.419Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2024-09-17T09:22:12.549 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[42985]: level=info ts=2024-09-17T09:22:12.419Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.29 port=9094 2024-09-17T09:22:12.549 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[42985]: level=info ts=2024-09-17T09:22:12.420Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T09:22:12.549 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[42985]: level=info ts=2024-09-17T09:22:12.440Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T09:22:12.549 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[42985]: level=info ts=2024-09-17T09:22:12.440Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T09:22:12.549 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[42985]: level=info ts=2024-09-17T09:22:12.443Z caller=main.go:497 msg=Listening address=:9093 2024-09-17T09:22:12.549 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 bash[42971]: a5a9a8d22f29626ff43a834df8d8be84037bb2a0f701de2c4968bf95ddaeabd7 2024-09-17T09:22:12.550 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:12 smithi029 systemd[1]: Started Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:22:12.830 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:10.589260+0000 mgr.y (mgr.14144) 216 : cluster [DBG] pgmap v202: 97 pgs: 4 creating+activating, 11 creating+peering, 29 unknown, 1 peering, 52 active+clean; 1.8 KiB data, 44 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-09-17T09:22:12.830 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:11.542969+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-09-17T09:22:12.830 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:11.546251+0000 mon.a (mon.0) 526 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T09:22:12.830 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:12.523758+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:13.932 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:12.524578+0000 mgr.y (mgr.14144) 217 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:22:13.932 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:12.526635+0000 mgr.y (mgr.14144) 218 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:22:13.932 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:12.546939+0000 mon.a (mon.0) 528 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-17T09:22:13.932 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:12.547089+0000 mon.a (mon.0) 529 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-09-17T09:22:13.932 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:12.549330+0000 mon.a (mon.0) 530 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:22:13.933 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:12.589584+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v205: 129 pgs: 4 creating+activating, 18 creating+peering, 28 unknown, 79 active+clean; 10 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 14 KiB/s wr, 57 op/s; 11 B/s, 0 objects/s recovering 2024-09-17T09:22:14.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:12.524578+0000 mgr.y (mgr.14144) 217 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:22:14.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:12.526635+0000 mgr.y (mgr.14144) 218 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:22:14.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:12.546939+0000 mon.a (mon.0) 528 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-17T09:22:14.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:12.547089+0000 mon.a (mon.0) 529 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-09-17T09:22:14.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:12.549330+0000 mon.a (mon.0) 530 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:22:14.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:12.589584+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v205: 129 pgs: 4 creating+activating, 18 creating+peering, 28 unknown, 79 active+clean; 10 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 14 KiB/s wr, 57 op/s; 11 B/s, 0 objects/s recovering 2024-09-17T09:22:14.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:12.524578+0000 mgr.y (mgr.14144) 217 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:22:14.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:12.526635+0000 mgr.y (mgr.14144) 218 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:22:14.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:12.546939+0000 mon.a (mon.0) 528 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-17T09:22:14.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:12.547089+0000 mon.a (mon.0) 529 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-09-17T09:22:14.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:12.549330+0000 mon.a (mon.0) 530 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T09:22:14.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:12.589584+0000 mgr.y (mgr.14144) 219 : cluster [DBG] pgmap v205: 129 pgs: 4 creating+activating, 18 creating+peering, 28 unknown, 79 active+clean; 10 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 14 KiB/s wr, 57 op/s; 11 B/s, 0 objects/s recovering 2024-09-17T09:22:14.653 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:22:14.653 INFO:teuthology.orchestra.run.smithi029.stdout: "id": "a8c81a13-82ba-4e43-8bb8-9b557c7ecf25", 2024-09-17T09:22:14.653 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 1, 2024-09-17T09:22:14.653 INFO:teuthology.orchestra.run.smithi029.stdout: "predecessor_uuid": "8aac02fd-4bc3-484e-b246-d0c65dbac374", 2024-09-17T09:22:14.653 INFO:teuthology.orchestra.run.smithi029.stdout: "sync_status": [], 2024-09-17T09:22:14.653 INFO:teuthology.orchestra.run.smithi029.stdout: "period_map": { 2024-09-17T09:22:14.653 INFO:teuthology.orchestra.run.smithi029.stdout: "id": "a8c81a13-82ba-4e43-8bb8-9b557c7ecf25", 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: "zonegroups": [ 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: "id": "bd3db493-dd05-4278-aa3c-59043c019dcb", 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default", 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: "api_name": "default", 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: "is_master": "true", 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: "endpoints": [], 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: "hostnames": [], 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: "hostnames_s3website": [], 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: "master_zone": "9380a79a-b931-4490-9bf2-22379873f915", 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: "zones": [ 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: "id": "9380a79a-b931-4490-9bf2-22379873f915", 2024-09-17T09:22:14.654 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "z", 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: "endpoints": [], 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: "log_meta": "false", 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: "log_data": "false", 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: "bucket_index_max_shards": 11, 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: "read_only": "false", 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_type": "", 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: "sync_from_all": "true", 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: "sync_from": [], 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: "redirect_zone": "" 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: "placement_targets": [ 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-17T09:22:14.655 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "default-placement", 2024-09-17T09:22:14.656 INFO:teuthology.orchestra.run.smithi029.stdout: "tags": [], 2024-09-17T09:22:14.656 INFO:teuthology.orchestra.run.smithi029.stdout: "storage_classes": [ 2024-09-17T09:22:14.656 INFO:teuthology.orchestra.run.smithi029.stdout: "STANDARD" 2024-09-17T09:22:14.656 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-17T09:22:14.656 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:22:14.656 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:22:14.656 INFO:teuthology.orchestra.run.smithi029.stdout: "default_placement": "default-placement", 2024-09-17T09:22:14.656 INFO:teuthology.orchestra.run.smithi029.stdout: "realm_id": "d084521a-8d82-4dc2-bdd6-3f665e8bd2be", 2024-09-17T09:22:14.656 INFO:teuthology.orchestra.run.smithi029.stdout: "sync_policy": { 2024-09-17T09:22:14.656 INFO:teuthology.orchestra.run.smithi029.stdout: "groups": [] 2024-09-17T09:22:14.656 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:22:14.656 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:22:14.656 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: "short_zone_ids": [ 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "9380a79a-b931-4490-9bf2-22379873f915", 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: "val": 3285284766 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: "master_zonegroup": "bd3db493-dd05-4278-aa3c-59043c019dcb", 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: "master_zone": "9380a79a-b931-4490-9bf2-22379873f915", 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: "period_config": { 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: "bucket_quota": { 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": false, 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: "check_on_raw": false, 2024-09-17T09:22:14.657 INFO:teuthology.orchestra.run.smithi029.stdout: "max_size": -1, 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: "max_size_kb": 0, 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: "max_objects": -1 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: "user_quota": { 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: "enabled": false, 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: "check_on_raw": false, 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: "max_size": -1, 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: "max_size_kb": 0, 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: "max_objects": -1 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: "realm_id": "d084521a-8d82-4dc2-bdd6-3f665e8bd2be", 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: "realm_name": "r", 2024-09-17T09:22:14.658 INFO:teuthology.orchestra.run.smithi029.stdout: "realm_epoch": 2 2024-09-17T09:22:14.659 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:22:14.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:13.553403+0000 mon.a (mon.0) 531 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T09:22:14.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:13.553493+0000 mon.a (mon.0) 532 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-09-17T09:22:14.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:13.554525+0000 mon.a (mon.0) 533 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-17T09:22:14.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[42985]: level=info ts=2024-09-17T09:22:14.421Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000086551s 2024-09-17T09:22:14.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:13.553403+0000 mon.a (mon.0) 531 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T09:22:14.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:13.553493+0000 mon.a (mon.0) 532 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-09-17T09:22:14.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:13.554525+0000 mon.a (mon.0) 533 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-17T09:22:15.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:13.553403+0000 mon.a (mon.0) 531 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T09:22:15.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:13.553493+0000 mon.a (mon.0) 532 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-09-17T09:22:15.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:13.554525+0000 mon.a (mon.0) 533 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-17T09:22:15.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-09-17T09:22:15.600 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:14.555032+0000 mon.a (mon.0) 534 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-09-17T09:22:15.600 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:14.555139+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-09-17T09:22:15.600 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:14.589909+0000 mgr.y (mgr.14144) 220 : cluster [DBG] pgmap v208: 129 pgs: 4 creating+activating, 18 creating+peering, 22 unknown, 85 active+clean; 12 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 17 KiB/s wr, 69 op/s; 11 B/s, 0 objects/s recovering 2024-09-17T09:22:15.601 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 systemd[1]: Stopping Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:22:15.601 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=warn ts=2024-09-17T09:22:15.589Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2024-09-17T09:22:15.601 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:22:15.589Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:22:15.601 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:22:15.589Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:22:15.601 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:22:15.589Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:22:15.601 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:22:15.589Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:22:15.601 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:22:15.589Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:22:15.601 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:22:15.589Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:22:15.602 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:22:15.590Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:22:15.602 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:22:15.590Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:22:15.602 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:22:15.590Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:22:15.602 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:22:15.590Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:22:15.602 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[35782]: level=info ts=2024-09-17T09:22:15.590Z caller=main.go:761 msg="See you next time!" 2024-09-17T09:22:15.602 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 podman[37086]: 2024-09-17 09:22:15.600000697 +0000 UTC m=+0.115015598 container died b995939d97369b77770a41cb6396bc3d9487fd3f97589bb3349515f4db506ced (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:22:15.743 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:14.555032+0000 mon.a (mon.0) 534 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-09-17T09:22:15.743 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:14.555139+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-09-17T09:22:15.743 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:14.589909+0000 mgr.y (mgr.14144) 220 : cluster [DBG] pgmap v208: 129 pgs: 4 creating+activating, 18 creating+peering, 22 unknown, 85 active+clean; 12 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 17 KiB/s wr, 69 op/s; 11 B/s, 0 objects/s recovering 2024-09-17T09:22:15.743 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:14.555032+0000 mon.a (mon.0) 534 : audit [INF] from='client.? 172.21.15.29:0/3297814818' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-09-17T09:22:15.743 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:14.555139+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-09-17T09:22:15.743 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:14.589909+0000 mgr.y (mgr.14144) 220 : cluster [DBG] pgmap v208: 129 pgs: 4 creating+activating, 18 creating+peering, 22 unknown, 85 active+clean; 12 KiB data, 46 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 17 KiB/s wr, 69 op/s; 11 B/s, 0 objects/s recovering 2024-09-17T09:22:16.182 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 podman[37086]: 2024-09-17 09:22:15.947142689 +0000 UTC m=+0.462157573 container cleanup b995939d97369b77770a41cb6396bc3d9487fd3f97589bb3349515f4db506ced (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:22:16.183 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:15 smithi103 podman[37086]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a 2024-09-17T09:22:16.505 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 podman[37098]: 2024-09-17 09:22:16.181718363 +0000 UTC m=+0.583183533 container remove b995939d97369b77770a41cb6396bc3d9487fd3f97589bb3349515f4db506ced (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:22:16.505 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T09:22:16.506 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:22:16.506 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:22:16.506 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 podman[37152]: 2024-09-17 09:22:16.261455269 +0000 UTC m=+0.017175670 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:22:16.506 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 podman[37152]: 2024-09-17 09:22:16.399378851 +0000 UTC m=+0.155099233 container create 94209ce0b68f7a5d633f4b2d0959e023b00803b892b553d735137e9a44ed7142 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:22:16.779 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 podman[37152]: 2024-09-17 09:22:16.626522558 +0000 UTC m=+0.382242948 container init 94209ce0b68f7a5d633f4b2d0959e023b00803b892b553d735137e9a44ed7142 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 podman[37152]: 2024-09-17 09:22:16.629408098 +0000 UTC m=+0.385128485 container start 94209ce0b68f7a5d633f4b2d0959e023b00803b892b553d735137e9a44ed7142 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.659Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.659Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.659Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.659Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.659Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.659Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.661Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.661Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.664Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.664Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.664Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.664Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=348.116µs 2024-09-17T09:22:16.780 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.665Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:22:16.781 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.665Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:22:16.781 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.665Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:22:16.781 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.666Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:22:16.781 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:22:16.666Z caller=main.go:646 msg="Server is ready to receive web requests." 2024-09-17T09:22:16.781 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 bash[37152]: 94209ce0b68f7a5d633f4b2d0959e023b00803b892b553d735137e9a44ed7142 2024-09-17T09:22:16.781 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:22:16 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:22:17.430 INFO:teuthology.orchestra.run.smithi029.stdout:Scheduled rgw.foo update... 2024-09-17T09:22:18.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:16.590540+0000 mgr.y (mgr.14144) 221 : cluster [DBG] pgmap v209: 129 pgs: 7 creating+peering, 122 active+clean; 25 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 28 KiB/s wr, 111 op/s 2024-09-17T09:22:18.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.754598+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.757551+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:22:18.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.757958+0000 mgr.y (mgr.14144) 222 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:22:18.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.759082+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029:9093"}]: dispatch 2024-09-17T09:22:18.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.759396+0000 mgr.y (mgr.14144) 223 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029:9093"}]: dispatch 2024-09-17T09:22:18.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.765177+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.770136+0000 mon.a (mon.0) 540 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:22:18.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.770551+0000 mgr.y (mgr.14144) 224 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:22:18.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.779451+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103:9095"}]: dispatch 2024-09-17T09:22:18.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.779824+0000 mgr.y (mgr.14144) 225 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103:9095"}]: dispatch 2024-09-17T09:22:18.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.784246+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.788491+0000 mon.a (mon.0) 543 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:22:18.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.788903+0000 mgr.y (mgr.14144) 226 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:22:18.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.793636+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103:3000"}]: dispatch 2024-09-17T09:22:18.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.794012+0000 mgr.y (mgr.14144) 227 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103:3000"}]: dispatch 2024-09-17T09:22:18.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:16.798883+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:17.429546+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.013 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:16.590540+0000 mgr.y (mgr.14144) 221 : cluster [DBG] pgmap v209: 129 pgs: 7 creating+peering, 122 active+clean; 25 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 28 KiB/s wr, 111 op/s 2024-09-17T09:22:18.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.754598+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.757551+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:22:18.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.757958+0000 mgr.y (mgr.14144) 222 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:22:18.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.759082+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029:9093"}]: dispatch 2024-09-17T09:22:18.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.759396+0000 mgr.y (mgr.14144) 223 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029:9093"}]: dispatch 2024-09-17T09:22:18.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.765177+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.770136+0000 mon.a (mon.0) 540 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:22:18.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.770551+0000 mgr.y (mgr.14144) 224 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:22:18.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.779451+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103:9095"}]: dispatch 2024-09-17T09:22:18.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.779824+0000 mgr.y (mgr.14144) 225 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103:9095"}]: dispatch 2024-09-17T09:22:18.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.784246+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.788491+0000 mon.a (mon.0) 543 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:22:18.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.788903+0000 mgr.y (mgr.14144) 226 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:22:18.015 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.793636+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103:3000"}]: dispatch 2024-09-17T09:22:18.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:16.590540+0000 mgr.y (mgr.14144) 221 : cluster [DBG] pgmap v209: 129 pgs: 7 creating+peering, 122 active+clean; 25 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 28 KiB/s wr, 111 op/s 2024-09-17T09:22:18.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.754598+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.757551+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:22:18.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.757958+0000 mgr.y (mgr.14144) 222 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:22:18.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.759082+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029:9093"}]: dispatch 2024-09-17T09:22:18.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.759396+0000 mgr.y (mgr.14144) 223 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029:9093"}]: dispatch 2024-09-17T09:22:18.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.765177+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.770136+0000 mon.a (mon.0) 540 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:22:18.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.770551+0000 mgr.y (mgr.14144) 224 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:22:18.016 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.779451+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103:9095"}]: dispatch 2024-09-17T09:22:18.016 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.779824+0000 mgr.y (mgr.14144) 225 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103:9095"}]: dispatch 2024-09-17T09:22:18.016 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.784246+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.016 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.788491+0000 mon.a (mon.0) 543 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:22:18.016 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.788903+0000 mgr.y (mgr.14144) 226 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:22:18.016 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.793636+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103:3000"}]: dispatch 2024-09-17T09:22:18.016 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.794012+0000 mgr.y (mgr.14144) 227 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103:3000"}]: dispatch 2024-09-17T09:22:18.016 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:16.798883+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.017 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:17.429546+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.017 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.794012+0000 mgr.y (mgr.14144) 227 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103:3000"}]: dispatch 2024-09-17T09:22:18.017 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:16.798883+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.017 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:17.429546+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:18.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch apply rgw smpl' 2024-09-17T09:22:19.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:17.425473+0000 mgr.y (mgr.14144) 228 : audit [DBG] from='client.14619 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:22:19.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:17.426373+0000 mgr.y (mgr.14144) 229 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2024-09-17T09:22:19.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:17.425473+0000 mgr.y (mgr.14144) 228 : audit [DBG] from='client.14619 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:22:19.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:17.426373+0000 mgr.y (mgr.14144) 229 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2024-09-17T09:22:19.171 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:17.425473+0000 mgr.y (mgr.14144) 228 : audit [DBG] from='client.14619 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:22:19.171 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:17.426373+0000 mgr.y (mgr.14144) 229 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2024-09-17T09:22:20.139 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:18.591170+0000 mgr.y (mgr.14144) 230 : cluster [DBG] pgmap v210: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 18 KiB/s wr, 71 op/s 2024-09-17T09:22:20.139 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:18.591170+0000 mgr.y (mgr.14144) 230 : cluster [DBG] pgmap v210: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 18 KiB/s wr, 71 op/s 2024-09-17T09:22:20.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:18.591170+0000 mgr.y (mgr.14144) 230 : cluster [DBG] pgmap v210: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 18 KiB/s wr, 71 op/s 2024-09-17T09:22:20.620 INFO:teuthology.orchestra.run.smithi029.stdout:Scheduled rgw.smpl update... 2024-09-17T09:22:21.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:20.071632+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:21.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:20.619532+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:21.250 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:20.071632+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:21.250 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:20.619532+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:21.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:20.071632+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:21.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:20.619532+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:21.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph osd pool create foo' 2024-09-17T09:22:22.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:20.591639+0000 mgr.y (mgr.14144) 231 : cluster [DBG] pgmap v211: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 14 KiB/s wr, 54 op/s 2024-09-17T09:22:22.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:20.615897+0000 mgr.y (mgr.14144) 232 : audit [DBG] from='client.14625 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:22:22.250 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:20.616361+0000 mgr.y (mgr.14144) 233 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2024-09-17T09:22:22.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:20.591639+0000 mgr.y (mgr.14144) 231 : cluster [DBG] pgmap v211: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 14 KiB/s wr, 54 op/s 2024-09-17T09:22:22.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:20.615897+0000 mgr.y (mgr.14144) 232 : audit [DBG] from='client.14625 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:22:22.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:20.616361+0000 mgr.y (mgr.14144) 233 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2024-09-17T09:22:22.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:20.591639+0000 mgr.y (mgr.14144) 231 : cluster [DBG] pgmap v211: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 9.9 KiB/s rd, 14 KiB/s wr, 54 op/s 2024-09-17T09:22:22.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:20.615897+0000 mgr.y (mgr.14144) 232 : audit [DBG] from='client.14625 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:22:22.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:20.616361+0000 mgr.y (mgr.14144) 233 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2024-09-17T09:22:22.548 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:22:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[42985]: level=info ts=2024-09-17T09:22:22.421Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000696472s 2024-09-17T09:22:24.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:22.592100+0000 mgr.y (mgr.14144) 234 : cluster [DBG] pgmap v212: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 11 KiB/s wr, 42 op/s 2024-09-17T09:22:24.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:22.912005+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:22.916136+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:22.922000+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:22.926567+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:22.927147+0000 mgr.y (mgr.14144) 235 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2024-09-17T09:22:24.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:22.930361+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:22.930913+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi029.tyykln", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:22:24.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:22.933968+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi029.tyykln", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:22:24.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:22.937673+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:22.938325+0000 mon.a (mon.0) 557 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:24.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:22.938931+0000 mgr.y (mgr.14144) 236 : cephadm [INF] Deploying daemon rgw.foo.smithi029.tyykln on smithi029 2024-09-17T09:22:24.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:22.592100+0000 mgr.y (mgr.14144) 234 : cluster [DBG] pgmap v212: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 11 KiB/s wr, 42 op/s 2024-09-17T09:22:24.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:22.912005+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:22.916136+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:22.922000+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:22.926567+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:22.927147+0000 mgr.y (mgr.14144) 235 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2024-09-17T09:22:24.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:22.930361+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:22.930913+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi029.tyykln", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:22:24.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:22.933968+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi029.tyykln", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:22:24.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:22.937673+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:22.938325+0000 mon.a (mon.0) 557 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:24.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:22.938931+0000 mgr.y (mgr.14144) 236 : cephadm [INF] Deploying daemon rgw.foo.smithi029.tyykln on smithi029 2024-09-17T09:22:24.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:22.592100+0000 mgr.y (mgr.14144) 234 : cluster [DBG] pgmap v212: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 11 KiB/s wr, 42 op/s 2024-09-17T09:22:24.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:22.912005+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:22.916136+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:22.922000+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:22.926567+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:22.927147+0000 mgr.y (mgr.14144) 235 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2024-09-17T09:22:24.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:22.930361+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:22.930913+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi029.tyykln", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:22:24.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:22.933968+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi029.tyykln", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:22:24.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:22.937673+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:24.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:22.938325+0000 mon.a (mon.0) 557 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:24.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:22.938931+0000 mgr.y (mgr.14144) 236 : cephadm [INF] Deploying daemon rgw.foo.smithi029.tyykln on smithi029 2024-09-17T09:22:26.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:24.592754+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v213: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 9.9 KiB/s wr, 38 op/s 2024-09-17T09:22:26.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:24.592754+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v213: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 9.9 KiB/s wr, 38 op/s 2024-09-17T09:22:26.059 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'foo' created 2024-09-17T09:22:26.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:24.592754+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v213: 129 pgs: 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 9.9 KiB/s wr, 38 op/s 2024-09-17T09:22:26.999 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:25.541917+0000 mon.c (mon.1) 18 : audit [INF] from='client.? 172.21.15.29:0/2137005706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T09:22:27.000 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:25.542279+0000 mon.a (mon.0) 558 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T09:22:27.000 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:26.046780+0000 mon.a (mon.0) 559 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-17T09:22:27.000 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:26.046861+0000 mon.a (mon.0) 560 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-17T09:22:27.000 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:26.351704+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:27.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:26.352599+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi103.xonepa", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:22:27.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:26.355961+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi103.xonepa", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:22:27.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:26.360360+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:27.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:26.361150+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:27.001 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:26.361902+0000 mgr.y (mgr.14144) 238 : cephadm [INF] Deploying daemon rgw.foo.smithi103.xonepa on smithi103 2024-09-17T09:22:27.002 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:25.541917+0000 mon.c (mon.1) 18 : audit [INF] from='client.? 172.21.15.29:0/2137005706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T09:22:27.002 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:25.542279+0000 mon.a (mon.0) 558 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T09:22:27.002 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:26.046780+0000 mon.a (mon.0) 559 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-17T09:22:27.002 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:26.046861+0000 mon.a (mon.0) 560 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-17T09:22:27.003 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:26.351704+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:27.003 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:26.352599+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi103.xonepa", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:22:27.003 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:26.355961+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi103.xonepa", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:22:27.003 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:26.360360+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:27.003 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:26.361150+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:27.003 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:26.361902+0000 mgr.y (mgr.14144) 238 : cephadm [INF] Deploying daemon rgw.foo.smithi103.xonepa on smithi103 2024-09-17T09:22:27.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'rbd pool init foo' 2024-09-17T09:22:27.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:25.541917+0000 mon.c (mon.1) 18 : audit [INF] from='client.? 172.21.15.29:0/2137005706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T09:22:27.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:25.542279+0000 mon.a (mon.0) 558 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T09:22:27.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:26.046780+0000 mon.a (mon.0) 559 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-17T09:22:27.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:26.046861+0000 mon.a (mon.0) 560 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-17T09:22:27.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:26.351704+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:27.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:26.352599+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi103.xonepa", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:22:27.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:26.355961+0000 mon.a (mon.0) 563 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi103.xonepa", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:22:27.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:26.360360+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:27.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:26.361150+0000 mon.a (mon.0) 565 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:27.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:26.361902+0000 mgr.y (mgr.14144) 238 : cephadm [INF] Deploying daemon rgw.foo.smithi103.xonepa on smithi103 2024-09-17T09:22:28.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:26.593151+0000 mgr.y (mgr.14144) 239 : cluster [DBG] pgmap v215: 161 pgs: 2 creating+peering, 30 unknown, 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 1.4 KiB/s wr, 5 op/s 2024-09-17T09:22:28.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:27.047859+0000 mon.a (mon.0) 566 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-17T09:22:28.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:26.593151+0000 mgr.y (mgr.14144) 239 : cluster [DBG] pgmap v215: 161 pgs: 2 creating+peering, 30 unknown, 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 1.4 KiB/s wr, 5 op/s 2024-09-17T09:22:28.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:27.047859+0000 mon.a (mon.0) 566 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-17T09:22:28.329 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:26.593151+0000 mgr.y (mgr.14144) 239 : cluster [DBG] pgmap v215: 161 pgs: 2 creating+peering, 30 unknown, 129 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 1.4 KiB/s wr, 5 op/s 2024-09-17T09:22:28.329 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:27.047859+0000 mon.a (mon.0) 566 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-17T09:22:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:28.051776+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-17T09:22:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:28.719386+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.29:0/2821781414' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T09:22:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:28.719736+0000 mon.a (mon.0) 568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T09:22:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:28.751552+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:28.755223+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:28.755799+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi103.gwwbkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:22:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:28.758776+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi103.gwwbkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:22:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:28.762669+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:28.763353+0000 mon.a (mon.0) 574 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:28.051776+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-17T09:22:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:28.719386+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.29:0/2821781414' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T09:22:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:28.719736+0000 mon.a (mon.0) 568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T09:22:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:28.751552+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:28.755223+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:28.755799+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi103.gwwbkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:22:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:28.758776+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi103.gwwbkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:22:29.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:28.762669+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:29.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:28.763353+0000 mon.a (mon.0) 574 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:29.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:28.051776+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-17T09:22:29.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:28.719386+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.29:0/2821781414' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T09:22:29.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:28.719736+0000 mon.a (mon.0) 568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T09:22:29.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:28.751552+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:29.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:28.755223+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:29.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:28.755799+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi103.gwwbkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:22:29.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:28.758776+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi103.gwwbkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:22:29.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:28.762669+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:29.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:28.763353+0000 mon.a (mon.0) 574 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:30.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:28.593521+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v218: 161 pgs: 7 creating+peering, 19 unknown, 135 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-17T09:22:30.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:28.752503+0000 mgr.y (mgr.14144) 241 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2024-09-17T09:22:30.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:28.763953+0000 mgr.y (mgr.14144) 242 : cephadm [INF] Deploying daemon rgw.smpl.smithi103.gwwbkl on smithi103 2024-09-17T09:22:30.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:29.058308+0000 mon.a (mon.0) 575 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-17T09:22:30.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:29.058431+0000 mon.a (mon.0) 576 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-17T09:22:30.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:28.593521+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v218: 161 pgs: 7 creating+peering, 19 unknown, 135 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-17T09:22:30.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:28.752503+0000 mgr.y (mgr.14144) 241 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2024-09-17T09:22:30.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:28.763953+0000 mgr.y (mgr.14144) 242 : cephadm [INF] Deploying daemon rgw.smpl.smithi103.gwwbkl on smithi103 2024-09-17T09:22:30.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:29.058308+0000 mon.a (mon.0) 575 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-17T09:22:30.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:29.058431+0000 mon.a (mon.0) 576 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-17T09:22:30.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:28.593521+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v218: 161 pgs: 7 creating+peering, 19 unknown, 135 active+clean; 27 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 142 op/s 2024-09-17T09:22:30.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:28.752503+0000 mgr.y (mgr.14144) 241 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2024-09-17T09:22:30.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:28.763953+0000 mgr.y (mgr.14144) 242 : cephadm [INF] Deploying daemon rgw.smpl.smithi103.gwwbkl on smithi103 2024-09-17T09:22:30.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:29.058308+0000 mon.a (mon.0) 575 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-17T09:22:30.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:29.058431+0000 mon.a (mon.0) 576 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-17T09:22:31.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:30.059041+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-17T09:22:31.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:31.058804+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:31.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:31.059821+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi029.prstdw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:22:31.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:30.059041+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-17T09:22:31.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:31.058804+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:31.514 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:31.059821+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi029.prstdw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:22:31.514 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:30.059041+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-17T09:22:31.515 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:31.058804+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:31.515 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:31.059821+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi029.prstdw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:22:32.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:30.594146+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v221: 161 pgs: 7 creating+peering, 3 unknown, 151 active+clean; 27 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 344 KiB/s rd, 255 B/s wr, 669 op/s 2024-09-17T09:22:32.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:31.064197+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi029.prstdw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:22:32.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:31.069520+0000 mon.a (mon.0) 581 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-17T09:22:32.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:31.076036+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:32.502 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:31.077427+0000 mon.a (mon.0) 583 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:32.502 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:31.078301+0000 mgr.y (mgr.14144) 244 : cephadm [INF] Deploying daemon rgw.smpl.smithi029.prstdw on smithi029 2024-09-17T09:22:32.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:30.594146+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v221: 161 pgs: 7 creating+peering, 3 unknown, 151 active+clean; 27 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 344 KiB/s rd, 255 B/s wr, 669 op/s 2024-09-17T09:22:32.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:31.064197+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi029.prstdw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:22:32.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:31.069520+0000 mon.a (mon.0) 581 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-17T09:22:32.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:31.076036+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:32.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:31.077427+0000 mon.a (mon.0) 583 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:32.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:31.078301+0000 mgr.y (mgr.14144) 244 : cephadm [INF] Deploying daemon rgw.smpl.smithi029.prstdw on smithi029 2024-09-17T09:22:32.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:30.594146+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v221: 161 pgs: 7 creating+peering, 3 unknown, 151 active+clean; 27 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 344 KiB/s rd, 255 B/s wr, 669 op/s 2024-09-17T09:22:32.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:31.064197+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi029.prstdw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T09:22:32.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:31.069520+0000 mon.a (mon.0) 581 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-17T09:22:32.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:31.076036+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:32.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:31.077427+0000 mon.a (mon.0) 583 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:32.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:31.078301+0000 mgr.y (mgr.14144) 244 : cephadm [INF] Deploying daemon rgw.smpl.smithi029.prstdw on smithi029 2024-09-17T09:22:32.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch apply iscsi foo u p' 2024-09-17T09:22:34.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:32.594708+0000 mgr.y (mgr.14144) 245 : cluster [DBG] pgmap v223: 161 pgs: 161 active+clean; 27 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 466 KiB/s rd, 675 B/s wr, 857 op/s 2024-09-17T09:22:34.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:32.594708+0000 mgr.y (mgr.14144) 245 : cluster [DBG] pgmap v223: 161 pgs: 161 active+clean; 27 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 466 KiB/s rd, 675 B/s wr, 857 op/s 2024-09-17T09:22:34.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:32.594708+0000 mgr.y (mgr.14144) 245 : cluster [DBG] pgmap v223: 161 pgs: 161 active+clean; 27 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 466 KiB/s rd, 675 B/s wr, 857 op/s 2024-09-17T09:22:35.260 INFO:teuthology.orchestra.run.smithi029.stdout:Scheduled iscsi.foo update... 2024-09-17T09:22:35.926 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'sleep 120' 2024-09-17T09:22:36.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:34.595071+0000 mgr.y (mgr.14144) 246 : cluster [DBG] pgmap v224: 161 pgs: 161 active+clean; 27 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 370 KiB/s rd, 511 B/s wr, 676 op/s 2024-09-17T09:22:36.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:35.253340+0000 mgr.y (mgr.14144) 247 : audit [DBG] from='client.24625 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:22:36.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:35.253864+0000 mgr.y (mgr.14144) 248 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2024-09-17T09:22:36.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:35.258028+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:36.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:35.329376+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:36.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:34.595071+0000 mgr.y (mgr.14144) 246 : cluster [DBG] pgmap v224: 161 pgs: 161 active+clean; 27 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 370 KiB/s rd, 511 B/s wr, 676 op/s 2024-09-17T09:22:36.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:35.253340+0000 mgr.y (mgr.14144) 247 : audit [DBG] from='client.24625 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:22:36.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:35.253864+0000 mgr.y (mgr.14144) 248 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2024-09-17T09:22:36.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:35.258028+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:36.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:35.329376+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:36.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:34.595071+0000 mgr.y (mgr.14144) 246 : cluster [DBG] pgmap v224: 161 pgs: 161 active+clean; 27 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 370 KiB/s rd, 511 B/s wr, 676 op/s 2024-09-17T09:22:36.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:35.253340+0000 mgr.y (mgr.14144) 247 : audit [DBG] from='client.24625 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:22:36.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:35.253864+0000 mgr.y (mgr.14144) 248 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2024-09-17T09:22:36.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:35.258028+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:36.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:35.329376+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:38.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:36.595739+0000 mgr.y (mgr.14144) 249 : cluster [DBG] pgmap v225: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 418 KiB/s rd, 814 B/s wr, 755 op/s 2024-09-17T09:22:38.463 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:38.024602+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-17T09:22:38.463 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:36.595739+0000 mgr.y (mgr.14144) 249 : cluster [DBG] pgmap v225: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 418 KiB/s rd, 814 B/s wr, 755 op/s 2024-09-17T09:22:38.464 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:38.024602+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-17T09:22:38.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:36.595739+0000 mgr.y (mgr.14144) 249 : cluster [DBG] pgmap v225: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 418 KiB/s rd, 814 B/s wr, 755 op/s 2024-09-17T09:22:38.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:38.024602+0000 mon.a (mon.0) 586 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-17T09:22:40.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:38.596151+0000 mgr.y (mgr.14144) 250 : cluster [DBG] pgmap v227: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 335 KiB/s rd, 639 B/s wr, 592 op/s 2024-09-17T09:22:40.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:38.710821+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:40.040 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:38.596151+0000 mgr.y (mgr.14144) 250 : cluster [DBG] pgmap v227: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 335 KiB/s rd, 639 B/s wr, 592 op/s 2024-09-17T09:22:40.040 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:38.710821+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:40.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:38.596151+0000 mgr.y (mgr.14144) 250 : cluster [DBG] pgmap v227: 161 pgs: 161 active+clean; 27 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 335 KiB/s rd, 639 B/s wr, 592 op/s 2024-09-17T09:22:40.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:38.710821+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:41.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:40.443238+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:41.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:40.446668+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:41.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:40.448808+0000 mgr.y (mgr.14144) 251 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2024-09-17T09:22:41.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:40.454281+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:22:41.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:40.457813+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T09:22:41.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:40.461398+0000 mon.a (mon.0) 592 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:41.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:40.462016+0000 mgr.y (mgr.14144) 252 : cephadm [INF] Deploying daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:22:41.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:40.596636+0000 mgr.y (mgr.14144) 253 : cluster [DBG] pgmap v228: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 295 KiB/s rd, 322 B/s wr, 515 op/s 2024-09-17T09:22:41.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:40.443238+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:41.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:40.446668+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:41.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:40.448808+0000 mgr.y (mgr.14144) 251 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2024-09-17T09:22:41.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:40.454281+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:22:41.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:40.457813+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T09:22:41.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:40.461398+0000 mon.a (mon.0) 592 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:41.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:40.462016+0000 mgr.y (mgr.14144) 252 : cephadm [INF] Deploying daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:22:41.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:40.596636+0000 mgr.y (mgr.14144) 253 : cluster [DBG] pgmap v228: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 295 KiB/s rd, 322 B/s wr, 515 op/s 2024-09-17T09:22:41.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:40.443238+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:41.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:40.446668+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:41.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:40.448808+0000 mgr.y (mgr.14144) 251 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2024-09-17T09:22:41.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:40.454281+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:22:41.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:40.457813+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T09:22:41.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:40.461398+0000 mon.a (mon.0) 592 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:22:41.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:40.462016+0000 mgr.y (mgr.14144) 252 : cephadm [INF] Deploying daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:22:41.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:40.596636+0000 mgr.y (mgr.14144) 253 : cluster [DBG] pgmap v228: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 295 KiB/s rd, 322 B/s wr, 515 op/s 2024-09-17T09:22:42.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:41.467903+0000 mon.a (mon.0) 593 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-09-17T09:22:42.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:41.467903+0000 mon.a (mon.0) 593 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-09-17T09:22:42.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:42 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:41.467903+0000 mon.a (mon.0) 593 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-09-17T09:22:44.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:42.597252+0000 mgr.y (mgr.14144) 254 : cluster [DBG] pgmap v229: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 201 KiB/s rd, 307 B/s wr, 350 op/s 2024-09-17T09:22:44.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:43.326724+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:44.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:43.769570+0000 mon.a (mon.0) 595 : audit [DBG] from='client.? 172.21.15.29:0/3397986490' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:22:44.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:44.065139+0000 mon.a (mon.0) 596 : audit [INF] from='client.? 172.21.15.29:0/2160103551' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:22:44.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:42.597252+0000 mgr.y (mgr.14144) 254 : cluster [DBG] pgmap v229: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 201 KiB/s rd, 307 B/s wr, 350 op/s 2024-09-17T09:22:44.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:43.326724+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:44.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:43.769570+0000 mon.a (mon.0) 595 : audit [DBG] from='client.? 172.21.15.29:0/3397986490' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:22:44.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:44.065139+0000 mon.a (mon.0) 596 : audit [INF] from='client.? 172.21.15.29:0/2160103551' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:22:44.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:42.597252+0000 mgr.y (mgr.14144) 254 : cluster [DBG] pgmap v229: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 201 KiB/s rd, 307 B/s wr, 350 op/s 2024-09-17T09:22:44.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:43.326724+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:44.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:43.769570+0000 mon.a (mon.0) 595 : audit [DBG] from='client.? 172.21.15.29:0/3397986490' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:22:44.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:44.065139+0000 mon.a (mon.0) 596 : audit [INF] from='client.? 172.21.15.29:0/2160103551' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:22:46.737 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:44.597849+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v230: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 183 KiB/s rd, 307 B/s wr, 318 op/s 2024-09-17T09:22:46.738 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:46 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:44.597849+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v230: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 183 KiB/s rd, 307 B/s wr, 318 op/s 2024-09-17T09:22:46.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:44.597849+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v230: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 183 KiB/s rd, 307 B/s wr, 318 op/s 2024-09-17T09:22:47.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:46.518202+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:47.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:46.598451+0000 mgr.y (mgr.14144) 256 : cluster [DBG] pgmap v231: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 102 B/s wr, 155 op/s 2024-09-17T09:22:47.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:46.518202+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:47.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:46.598451+0000 mgr.y (mgr.14144) 256 : cluster [DBG] pgmap v231: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 102 B/s wr, 155 op/s 2024-09-17T09:22:48.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:46.518202+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:48.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:46.598451+0000 mgr.y (mgr.14144) 256 : cluster [DBG] pgmap v231: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 102 B/s wr, 155 op/s 2024-09-17T09:22:49.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:47.862435+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:49.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:47.863504+0000 mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v232: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 104 B/s wr, 87 op/s 2024-09-17T09:22:49.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:47.866015+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:49.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:47.870452+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:22:49.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:47.870902+0000 mgr.y (mgr.14144) 258 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:22:49.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:22:47.872555+0000 mgr.y (mgr.14144) 259 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.29:5000 to Dashboard 2024-09-17T09:22:49.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:47.872751+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:22:49.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:47.873017+0000 mgr.y (mgr.14144) 260 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:22:49.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:47.876793+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:49.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:47.878827+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi029"}]: dispatch 2024-09-17T09:22:49.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:47.879248+0000 mgr.y (mgr.14144) 261 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi029"}]: dispatch 2024-09-17T09:22:49.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:47.885000+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:49.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:47.862435+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:49.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:47.863504+0000 mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v232: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 104 B/s wr, 87 op/s 2024-09-17T09:22:49.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:47.866015+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:49.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:47.870452+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:22:49.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:47.870902+0000 mgr.y (mgr.14144) 258 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:22:49.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:22:47.872555+0000 mgr.y (mgr.14144) 259 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.29:5000 to Dashboard 2024-09-17T09:22:49.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:47.872751+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:22:49.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:47.873017+0000 mgr.y (mgr.14144) 260 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:22:49.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:47.876793+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:49.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:47.878827+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi029"}]: dispatch 2024-09-17T09:22:49.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:47.879248+0000 mgr.y (mgr.14144) 261 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi029"}]: dispatch 2024-09-17T09:22:49.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:47.885000+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:49.207 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:47.862435+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:49.208 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:47.863504+0000 mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v232: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 104 B/s wr, 87 op/s 2024-09-17T09:22:49.208 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:47.866015+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:49.208 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:47.870452+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:22:49.208 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:47.870902+0000 mgr.y (mgr.14144) 258 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:22:49.208 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:22:47.872555+0000 mgr.y (mgr.14144) 259 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.29:5000 to Dashboard 2024-09-17T09:22:49.209 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:47.872751+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:22:49.209 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:47.873017+0000 mgr.y (mgr.14144) 260 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:22:49.209 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:47.876793+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:49.209 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:47.878827+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi029"}]: dispatch 2024-09-17T09:22:49.210 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:47.879248+0000 mgr.y (mgr.14144) 261 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi029"}]: dispatch 2024-09-17T09:22:49.210 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:47.885000+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:50.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:48.860754+0000 mon.a (mon.0) 605 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:22:50.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:48.860754+0000 mon.a (mon.0) 605 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:22:50.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:48.860754+0000 mon.a (mon.0) 605 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:22:51.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:49.863986+0000 mgr.y (mgr.14144) 262 : cluster [DBG] pgmap v233: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 90 B/s wr, 62 op/s 2024-09-17T09:22:51.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:49.863986+0000 mgr.y (mgr.14144) 262 : cluster [DBG] pgmap v233: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 90 B/s wr, 62 op/s 2024-09-17T09:22:51.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:49.863986+0000 mgr.y (mgr.14144) 262 : cluster [DBG] pgmap v233: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 90 B/s wr, 62 op/s 2024-09-17T09:22:52.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:50.910158+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:52.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:51.192321+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:52.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:51.195712+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:52.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:50.910158+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:52.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:51.192321+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:52.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:51.195712+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:52.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:50.910158+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:52.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:51.192321+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:52.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:51.195712+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:22:53.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:51.864887+0000 mgr.y (mgr.14144) 263 : cluster [DBG] pgmap v234: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 90 B/s rd, 90 B/s wr, 0 op/s 2024-09-17T09:22:53.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:51.864887+0000 mgr.y (mgr.14144) 263 : cluster [DBG] pgmap v234: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 90 B/s rd, 90 B/s wr, 0 op/s 2024-09-17T09:22:53.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:51.864887+0000 mgr.y (mgr.14144) 263 : cluster [DBG] pgmap v234: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 90 B/s rd, 90 B/s wr, 0 op/s 2024-09-17T09:22:55.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:53.865460+0000 mgr.y (mgr.14144) 264 : cluster [DBG] pgmap v235: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 90 B/s rd, 90 B/s wr, 0 op/s 2024-09-17T09:22:55.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:53.865460+0000 mgr.y (mgr.14144) 264 : cluster [DBG] pgmap v235: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 90 B/s rd, 90 B/s wr, 0 op/s 2024-09-17T09:22:55.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:53.865460+0000 mgr.y (mgr.14144) 264 : cluster [DBG] pgmap v235: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 90 B/s rd, 90 B/s wr, 0 op/s 2024-09-17T09:22:57.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:55.866178+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v236: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 90 B/s rd, 90 B/s wr, 0 op/s 2024-09-17T09:22:57.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:56.953077+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.29:0/509105262' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:22:57.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:55.866178+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v236: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 90 B/s rd, 90 B/s wr, 0 op/s 2024-09-17T09:22:57.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:56.953077+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.29:0/509105262' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:22:57.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:55.866178+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v236: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 90 B/s rd, 90 B/s wr, 0 op/s 2024-09-17T09:22:57.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:56.953077+0000 mon.c (mon.1) 20 : audit [DBG] from='client.? 172.21.15.29:0/509105262' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:22:58.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:22:57.244859+0000 mon.c (mon.1) 21 : audit [INF] from='client.? 172.21.15.29:0/2292510669' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:22:58.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:22:57.244859+0000 mon.c (mon.1) 21 : audit [INF] from='client.? 172.21.15.29:0/2292510669' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:22:58.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:22:57.244859+0000 mon.c (mon.1) 21 : audit [INF] from='client.? 172.21.15.29:0/2292510669' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:22:59.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:22:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:57.866912+0000 mgr.y (mgr.14144) 266 : cluster [DBG] pgmap v237: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:22:59.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:22:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:57.866912+0000 mgr.y (mgr.14144) 266 : cluster [DBG] pgmap v237: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:22:59.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:22:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:57.866912+0000 mgr.y (mgr.14144) 266 : cluster [DBG] pgmap v237: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:01.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:22:59.867565+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v238: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:22:59.867565+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v238: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:01.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:22:59.867565+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v238: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:03.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:01.868454+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v239: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:03.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:01.868454+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v239: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:01.868454+0000 mgr.y (mgr.14144) 268 : cluster [DBG] pgmap v239: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:03.869103+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v240: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:04.674842+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:23:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:04.676122+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:23:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:05.212990+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-17T09:23:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:05.213276+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-17T09:23:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:05.213483+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-09-17T09:23:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:03.869103+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v240: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:04.674842+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:23:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:04.676122+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:23:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:05.212990+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-17T09:23:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:05.213276+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-17T09:23:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:05.213483+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-09-17T09:23:05.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:03.869103+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v240: 161 pgs: 161 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:04.674842+0000 mon.a (mon.0) 609 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:23:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:04.676122+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:23:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:05.212990+0000 mon.a (mon.0) 611 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-17T09:23:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:05.213276+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-17T09:23:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:05.213483+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]: dispatch 2024-09-17T09:23:06.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:05.228282+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-17T09:23:06.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:05.228387+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-17T09:23:06.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:05.228464+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-17T09:23:06.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:05.228550+0000 mon.a (mon.0) 617 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-17T09:23:06.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:05.228282+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-17T09:23:06.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:05.228387+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-17T09:23:06.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:05.228464+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-17T09:23:06.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:05.228550+0000 mon.a (mon.0) 617 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-17T09:23:06.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:05.228282+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-17T09:23:06.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:05.228387+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-17T09:23:06.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:05.228464+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 5]}]': finished 2024-09-17T09:23:06.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:05.228550+0000 mon.a (mon.0) 617 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-17T09:23:07.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:05.869676+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v242: 161 pgs: 1 peering, 160 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T09:23:07.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:06.234156+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-17T09:23:07.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:05.869676+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v242: 161 pgs: 1 peering, 160 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T09:23:07.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:06.234156+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-17T09:23:07.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:05.869676+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v242: 161 pgs: 1 peering, 160 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T09:23:07.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:06.234156+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-17T09:23:09.433 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:07.870439+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v244: 161 pgs: 1 peering, 160 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T09:23:09.433 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:07.870439+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v244: 161 pgs: 1 peering, 160 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T09:23:09.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:07.870439+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v244: 161 pgs: 1 peering, 160 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T09:23:10.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:10.230935+0000 mon.c (mon.1) 22 : audit [DBG] from='client.? 172.21.15.29:0/4181460566' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:23:10.529 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:10.230935+0000 mon.c (mon.1) 22 : audit [DBG] from='client.? 172.21.15.29:0/4181460566' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:23:10.529 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:10.230935+0000 mon.c (mon.1) 22 : audit [DBG] from='client.? 172.21.15.29:0/4181460566' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:23:11.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:09.870958+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v245: 161 pgs: 1 peering, 160 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:11.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:10.525081+0000 mon.a (mon.0) 619 : audit [INF] from='client.? 172.21.15.29:0/4048296408' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:23:11.547 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:09.870958+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v245: 161 pgs: 1 peering, 160 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:11.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:10.525081+0000 mon.a (mon.0) 619 : audit [INF] from='client.? 172.21.15.29:0/4048296408' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:23:11.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:09.870958+0000 mgr.y (mgr.14144) 272 : cluster [DBG] pgmap v245: 161 pgs: 1 peering, 160 active+clean; 27 KiB data, 88 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:11.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:10.525081+0000 mon.a (mon.0) 619 : audit [INF] from='client.? 172.21.15.29:0/4048296408' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:23:13.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:11.871631+0000 mgr.y (mgr.14144) 273 : cluster [DBG] pgmap v246: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:13.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:11.871631+0000 mgr.y (mgr.14144) 273 : cluster [DBG] pgmap v246: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:13.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:11.871631+0000 mgr.y (mgr.14144) 273 : cluster [DBG] pgmap v246: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:15.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:13.872264+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v247: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 118 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:15.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:13.872264+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v247: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 118 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:15.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:13.872264+0000 mgr.y (mgr.14144) 274 : cluster [DBG] pgmap v247: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 118 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:17.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:15.873025+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v248: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:17.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:15.873025+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v248: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:17.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:15.873025+0000 mgr.y (mgr.14144) 275 : cluster [DBG] pgmap v248: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:19.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:17.873810+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v249: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 87 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:19.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:17.873810+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v249: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 87 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:19.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:17.873810+0000 mgr.y (mgr.14144) 276 : cluster [DBG] pgmap v249: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 87 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:21.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:19.874310+0000 mgr.y (mgr.14144) 277 : cluster [DBG] pgmap v250: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:21.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:19.874310+0000 mgr.y (mgr.14144) 277 : cluster [DBG] pgmap v250: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:21.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:19.874310+0000 mgr.y (mgr.14144) 277 : cluster [DBG] pgmap v250: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:23.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:21.875050+0000 mgr.y (mgr.14144) 278 : cluster [DBG] pgmap v251: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:23.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:21.875050+0000 mgr.y (mgr.14144) 278 : cluster [DBG] pgmap v251: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:23.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:21.875050+0000 mgr.y (mgr.14144) 278 : cluster [DBG] pgmap v251: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T09:23:24.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:23.723877+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.29:0/1435405263' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:23:24.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:24.016440+0000 mon.a (mon.0) 620 : audit [INF] from='client.? 172.21.15.29:0/2678001146' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:23:24.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:23.723877+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.29:0/1435405263' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:23:24.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:24.016440+0000 mon.a (mon.0) 620 : audit [INF] from='client.? 172.21.15.29:0/2678001146' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:23:24.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:23.723877+0000 mon.c (mon.1) 23 : audit [DBG] from='client.? 172.21.15.29:0/1435405263' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:23:24.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:24.016440+0000 mon.a (mon.0) 620 : audit [INF] from='client.? 172.21.15.29:0/2678001146' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:23:25.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:23.875506+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v252: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:25.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:23.875506+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v252: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:25.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:23.875506+0000 mgr.y (mgr.14144) 279 : cluster [DBG] pgmap v252: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:27.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:25.876346+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v253: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:27.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:25.876346+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v253: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:27.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:25.876346+0000 mgr.y (mgr.14144) 280 : cluster [DBG] pgmap v253: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:29.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:27.876982+0000 mgr.y (mgr.14144) 281 : cluster [DBG] pgmap v254: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:27.876982+0000 mgr.y (mgr.14144) 281 : cluster [DBG] pgmap v254: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:29.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:27.876982+0000 mgr.y (mgr.14144) 281 : cluster [DBG] pgmap v254: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:31.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:29.877530+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v255: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:31.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:29.877530+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v255: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:31.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:29.877530+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v255: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:33.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:31.878273+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v256: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:33.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:31.878273+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v256: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:33.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:31.878273+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v256: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:35.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:33.878981+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v257: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:35.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:33.878981+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v257: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:35.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:33.878981+0000 mgr.y (mgr.14144) 284 : cluster [DBG] pgmap v257: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:37.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:35.879594+0000 mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v258: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:37.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:35.879594+0000 mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v258: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:37.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:35.879594+0000 mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v258: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:38.542 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:37.302113+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.29:0/1552948863' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:23:38.542 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:37.592161+0000 mon.c (mon.1) 25 : audit [INF] from='client.? 172.21.15.29:0/520606847' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:23:38.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:37.302113+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.29:0/1552948863' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:23:38.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:37.592161+0000 mon.c (mon.1) 25 : audit [INF] from='client.? 172.21.15.29:0/520606847' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:23:38.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:37.302113+0000 mon.c (mon.1) 24 : audit [DBG] from='client.? 172.21.15.29:0/1552948863' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:23:38.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:37.592161+0000 mon.c (mon.1) 25 : audit [INF] from='client.? 172.21.15.29:0/520606847' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: access denied 2024-09-17T09:23:39.547 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:37.880283+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v259: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:39.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:37.880283+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v259: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:39.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:37.880283+0000 mgr.y (mgr.14144) 286 : cluster [DBG] pgmap v259: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:41.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:39.880885+0000 mgr.y (mgr.14144) 287 : cluster [DBG] pgmap v260: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:41.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:39.880885+0000 mgr.y (mgr.14144) 287 : cluster [DBG] pgmap v260: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:41.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:39.880885+0000 mgr.y (mgr.14144) 287 : cluster [DBG] pgmap v260: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:43.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:41.881630+0000 mgr.y (mgr.14144) 288 : cluster [DBG] pgmap v261: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:43.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:41.881630+0000 mgr.y (mgr.14144) 288 : cluster [DBG] pgmap v261: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:43.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:41.881630+0000 mgr.y (mgr.14144) 288 : cluster [DBG] pgmap v261: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:45.558 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:43.882251+0000 mgr.y (mgr.14144) 289 : cluster [DBG] pgmap v262: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:45.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:43.882251+0000 mgr.y (mgr.14144) 289 : cluster [DBG] pgmap v262: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:45.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:43.882251+0000 mgr.y (mgr.14144) 289 : cluster [DBG] pgmap v262: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:47.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:45.883021+0000 mgr.y (mgr.14144) 290 : cluster [DBG] pgmap v263: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:47.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:45.883021+0000 mgr.y (mgr.14144) 290 : cluster [DBG] pgmap v263: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:47.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:45.883021+0000 mgr.y (mgr.14144) 290 : cluster [DBG] pgmap v263: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:49.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:47.883753+0000 mgr.y (mgr.14144) 291 : cluster [DBG] pgmap v264: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:49.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:47.883753+0000 mgr.y (mgr.14144) 291 : cluster [DBG] pgmap v264: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:49.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:47.883753+0000 mgr.y (mgr.14144) 291 : cluster [DBG] pgmap v264: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:51.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:49.884329+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v265: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:51.712 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:49.884329+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v265: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:51.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:49.884329+0000 mgr.y (mgr.14144) 292 : cluster [DBG] pgmap v265: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:53.582 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:51.885178+0000 mgr.y (mgr.14144) 293 : cluster [DBG] pgmap v266: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:53.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:51.885178+0000 mgr.y (mgr.14144) 293 : cluster [DBG] pgmap v266: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:53.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:51.885178+0000 mgr.y (mgr.14144) 293 : cluster [DBG] pgmap v266: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:23:55.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:53.885757+0000 mgr.y (mgr.14144) 294 : cluster [DBG] pgmap v267: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:55.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:54.004438+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:23:55.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:54.220481+0000 mgr.y (mgr.14144) 295 : cluster [DBG] pgmap v268: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:55.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:23:54.224428+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:23:55.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:53.885757+0000 mgr.y (mgr.14144) 294 : cluster [DBG] pgmap v267: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:55.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:54.004438+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:23:55.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:54.220481+0000 mgr.y (mgr.14144) 295 : cluster [DBG] pgmap v268: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:55.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:23:54.224428+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:23:55.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:53.885757+0000 mgr.y (mgr.14144) 294 : cluster [DBG] pgmap v267: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:55.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:54.004438+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:23:55.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:54.220481+0000 mgr.y (mgr.14144) 295 : cluster [DBG] pgmap v268: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:55.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:23:54.224428+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:23:57.530 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:56.221209+0000 mgr.y (mgr.14144) 296 : cluster [DBG] pgmap v269: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:57.531 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:56.221209+0000 mgr.y (mgr.14144) 296 : cluster [DBG] pgmap v269: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:57.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:56.221209+0000 mgr.y (mgr.14144) 296 : cluster [DBG] pgmap v269: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:59.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:23:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:23:58.221959+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v270: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:59.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:23:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:23:58.221959+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v270: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:23:59.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:23:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:23:58.221959+0000 mgr.y (mgr.14144) 297 : cluster [DBG] pgmap v270: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:00.222528+0000 mgr.y (mgr.14144) 298 : cluster [DBG] pgmap v271: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:01.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:00.222528+0000 mgr.y (mgr.14144) 298 : cluster [DBG] pgmap v271: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:01.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:00.222528+0000 mgr.y (mgr.14144) 298 : cluster [DBG] pgmap v271: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:03.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:02.223240+0000 mgr.y (mgr.14144) 299 : cluster [DBG] pgmap v272: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:02.223240+0000 mgr.y (mgr.14144) 299 : cluster [DBG] pgmap v272: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:03.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:02.223240+0000 mgr.y (mgr.14144) 299 : cluster [DBG] pgmap v272: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:05.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:04.223962+0000 mgr.y (mgr.14144) 300 : cluster [DBG] pgmap v273: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:05.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:04.680486+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:24:05.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:04.680745+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:24:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:04.223962+0000 mgr.y (mgr.14144) 300 : cluster [DBG] pgmap v273: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:04.680486+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:24:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:04.680745+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:24:05.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:04.223962+0000 mgr.y (mgr.14144) 300 : cluster [DBG] pgmap v273: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:05.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:04.680486+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:24:05.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:04.680745+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:24:07.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:06.224811+0000 mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v274: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:07.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:06.224811+0000 mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v274: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:07.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:06.224811+0000 mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v274: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:09.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:08.225404+0000 mgr.y (mgr.14144) 302 : cluster [DBG] pgmap v275: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:09.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:08.225404+0000 mgr.y (mgr.14144) 302 : cluster [DBG] pgmap v275: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:09.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:08.225404+0000 mgr.y (mgr.14144) 302 : cluster [DBG] pgmap v275: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:11.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:10.226039+0000 mgr.y (mgr.14144) 303 : cluster [DBG] pgmap v276: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:11.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:10.226039+0000 mgr.y (mgr.14144) 303 : cluster [DBG] pgmap v276: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:11.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:10.226039+0000 mgr.y (mgr.14144) 303 : cluster [DBG] pgmap v276: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:13.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:12.226677+0000 mgr.y (mgr.14144) 304 : cluster [DBG] pgmap v277: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:13.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:12.226677+0000 mgr.y (mgr.14144) 304 : cluster [DBG] pgmap v277: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:13.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:12.226677+0000 mgr.y (mgr.14144) 304 : cluster [DBG] pgmap v277: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:15.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:14.227266+0000 mgr.y (mgr.14144) 305 : cluster [DBG] pgmap v278: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:15.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:14.227266+0000 mgr.y (mgr.14144) 305 : cluster [DBG] pgmap v278: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:15.557 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:14.227266+0000 mgr.y (mgr.14144) 305 : cluster [DBG] pgmap v278: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:17.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:16.227913+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v279: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:17.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:16.227913+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v279: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:17.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:16.227913+0000 mgr.y (mgr.14144) 306 : cluster [DBG] pgmap v279: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:19.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:18.228512+0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v280: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:19.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:18.228512+0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v280: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:19.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:18.228512+0000 mgr.y (mgr.14144) 307 : cluster [DBG] pgmap v280: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:21.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:20.229120+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v281: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:21.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:20.229120+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v281: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:21.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:20.229120+0000 mgr.y (mgr.14144) 308 : cluster [DBG] pgmap v281: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:23.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:22.230010+0000 mgr.y (mgr.14144) 309 : cluster [DBG] pgmap v282: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:23.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:22.230010+0000 mgr.y (mgr.14144) 309 : cluster [DBG] pgmap v282: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:23.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:22.230010+0000 mgr.y (mgr.14144) 309 : cluster [DBG] pgmap v282: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:25.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:24.230488+0000 mgr.y (mgr.14144) 310 : cluster [DBG] pgmap v283: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:25.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:24.230488+0000 mgr.y (mgr.14144) 310 : cluster [DBG] pgmap v283: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:25.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:24.230488+0000 mgr.y (mgr.14144) 310 : cluster [DBG] pgmap v283: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:27.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:26.231083+0000 mgr.y (mgr.14144) 311 : cluster [DBG] pgmap v284: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:27.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:26.231083+0000 mgr.y (mgr.14144) 311 : cluster [DBG] pgmap v284: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:27.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:26.231083+0000 mgr.y (mgr.14144) 311 : cluster [DBG] pgmap v284: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:29.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:28.231754+0000 mgr.y (mgr.14144) 312 : cluster [DBG] pgmap v285: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:29.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:28.231754+0000 mgr.y (mgr.14144) 312 : cluster [DBG] pgmap v285: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:28.231754+0000 mgr.y (mgr.14144) 312 : cluster [DBG] pgmap v285: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:31.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:30.232390+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v286: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:31.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:30.232390+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v286: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:31.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:30.232390+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v286: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:33.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:32.233011+0000 mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v287: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:33.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:32.233011+0000 mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v287: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:33.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:32.233011+0000 mgr.y (mgr.14144) 314 : cluster [DBG] pgmap v287: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:35.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:34.233706+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v288: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:35.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:34.233706+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v288: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:35.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:34.233706+0000 mgr.y (mgr.14144) 315 : cluster [DBG] pgmap v288: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:37.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:36.234475+0000 mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v289: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:37.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:36.234475+0000 mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v289: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:37.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:36.234475+0000 mgr.y (mgr.14144) 316 : cluster [DBG] pgmap v289: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:39.239 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:38.235210+0000 mgr.y (mgr.14144) 317 : cluster [DBG] pgmap v290: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:39.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:38.235210+0000 mgr.y (mgr.14144) 317 : cluster [DBG] pgmap v290: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:39.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-17T09:24:39.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:38.235210+0000 mgr.y (mgr.14144) 317 : cluster [DBG] pgmap v290: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:41.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:40.235827+0000 mgr.y (mgr.14144) 318 : cluster [DBG] pgmap v291: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:41.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:40.235827+0000 mgr.y (mgr.14144) 318 : cluster [DBG] pgmap v291: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:41.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:40.235827+0000 mgr.y (mgr.14144) 318 : cluster [DBG] pgmap v291: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:41.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-17T09:24:42.508 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:41.315001+0000 mon.a (mon.0) 625 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T09:24:42.508 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:41.315001+0000 mon.a (mon.0) 625 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T09:24:42.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:42 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:41.315001+0000 mon.a (mon.0) 625 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T09:24:43.491 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:42.236376+0000 mgr.y (mgr.14144) 319 : cluster [DBG] pgmap v292: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:43.491 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:42.236376+0000 mgr.y (mgr.14144) 319 : cluster [DBG] pgmap v292: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:43.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:42.236376+0000 mgr.y (mgr.14144) 319 : cluster [DBG] pgmap v292: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:44.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-17T09:24:45.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:43.903487+0000 mon.a (mon.0) 626 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T09:24:45.060 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:44.236956+0000 mgr.y (mgr.14144) 320 : cluster [DBG] pgmap v293: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:45.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:43.903487+0000 mon.a (mon.0) 626 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T09:24:45.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:44.236956+0000 mgr.y (mgr.14144) 320 : cluster [DBG] pgmap v293: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:45.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:43.903487+0000 mon.a (mon.0) 626 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T09:24:45.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:44.236956+0000 mgr.y (mgr.14144) 320 : cluster [DBG] pgmap v293: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:47.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-17T09:24:47.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:46.237588+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v294: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:47.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:46.508460+0000 mon.a (mon.0) 627 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T09:24:47.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:46.516475+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:24:47.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:46.237588+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v294: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:47.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:46.508460+0000 mon.a (mon.0) 627 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T09:24:47.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:46.516475+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:24:47.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:46.237588+0000 mgr.y (mgr.14144) 321 : cluster [DBG] pgmap v294: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:47.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:46.508460+0000 mon.a (mon.0) 627 : audit [INF] from='client.? ' entity='client.admin' 2024-09-17T09:24:47.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:46.516475+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:24:49.212 INFO:teuthology.orchestra.run.smithi029.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:24:49.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:48.238136+0000 mgr.y (mgr.14144) 322 : cluster [DBG] pgmap v295: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:49.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:48.238136+0000 mgr.y (mgr.14144) 322 : cluster [DBG] pgmap v295: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:49.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:48.238136+0000 mgr.y (mgr.14144) 322 : cluster [DBG] pgmap v295: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:50.080 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T09:24:50.090 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-09-17T09:24:50.090 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-09-17T09:24:50.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:49.206891+0000 mgr.y (mgr.14144) 323 : audit [DBG] from='client.14877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:50.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:24:49.207468+0000 mgr.y (mgr.14144) 324 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:24:50.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:49.210978+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:24:50.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:49.216017+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:24:50.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:24:49.218739+0000 mgr.y (mgr.14144) 325 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:24:50.444 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:49.206891+0000 mgr.y (mgr.14144) 323 : audit [DBG] from='client.14877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:50.444 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:24:49.207468+0000 mgr.y (mgr.14144) 324 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:24:50.444 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:49.210978+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:24:50.445 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:49.216017+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:24:50.445 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:24:49.218739+0000 mgr.y (mgr.14144) 325 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:24:50.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:49.206891+0000 mgr.y (mgr.14144) 323 : audit [DBG] from='client.14877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:50.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:24:49.207468+0000 mgr.y (mgr.14144) 324 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:24:50.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:49.210978+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:24:50.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:49.216017+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:24:50.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:24:49.218739+0000 mgr.y (mgr.14144) 325 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:24:51.478 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:50.238585+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v296: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:51.478 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:50.238585+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v296: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:51.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:50.238585+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v296: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:52.379 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:24:53.214 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID 2024-09-17T09:24:53.214 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 running (2m) 59s ago 3m *:9093 *:9094 0.20.0 0881eb8f169f a5a9a8d22f29 2024-09-17T09:24:53.214 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 running (2m) 2m ago 2m *:3000 6.7.4 557c83e11646 b74e78c8a6b5 2024-09-17T09:24:53.214 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 error 59s ago 2m - 2024-09-17T09:24:53.214 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 running (7m) 2m ago 6m *:8443 16.2.0 24ecd6d5f14c 1a97fa2150f2 2024-09-17T09:24:53.215 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 running (8m) 59s ago 8m *:9283 16.2.0 24ecd6d5f14c 9d130c67b36d 2024-09-17T09:24:53.215 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (8m) 59s ago 8m - 16.2.0 24ecd6d5f14c f3476652dfdd 2024-09-17T09:24:53.215 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (7m) 2m ago 7m - 16.2.0 24ecd6d5f14c 203d1f26d7b8 2024-09-17T09:24:53.215 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (7m) 59s ago 7m - 16.2.0 24ecd6d5f14c aa05f3e5656a 2024-09-17T09:24:53.215 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 running (3m) 59s ago 3m *:9100 0.18.1 e5a616e4b9cf 8202f7eb3e96 2024-09-17T09:24:53.215 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 running (3m) 2m ago 3m *:9100 0.18.1 e5a616e4b9cf 0e6931583242 2024-09-17T09:24:53.215 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (6m) 59s ago 6m - 16.2.0 24ecd6d5f14c 991731ce08b5 2024-09-17T09:24:53.215 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (6m) 59s ago 6m - 16.2.0 24ecd6d5f14c d5bce63fadc8 2024-09-17T09:24:53.215 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (6m) 59s ago 6m - 16.2.0 24ecd6d5f14c e9678d2f1685 2024-09-17T09:24:53.215 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (5m) 59s ago 5m - 16.2.0 24ecd6d5f14c dcf7122a7d99 2024-09-17T09:24:53.215 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (5m) 2m ago 5m - 16.2.0 24ecd6d5f14c 048c13025d32 2024-09-17T09:24:53.215 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (5m) 2m ago 5m - 16.2.0 24ecd6d5f14c 10a64919fbc9 2024-09-17T09:24:53.215 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (4m) 2m ago 4m - 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:24:53.216 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (4m) 2m ago 4m - 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:24:53.216 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 running (2m) 2m ago 3m *:9095 2.18.1 de242295e225 94209ce0b68f 2024-09-17T09:24:53.216 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 running (2m) 59s ago 2m *:8000 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:24:53.216 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 running (2m) 2m ago 2m *:8000 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:24:53.216 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 running (2m) 59s ago 2m *:80 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:24:53.216 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 running (2m) 2m ago 2m *:80 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:24:53.547 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:52.239235+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v297: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:53.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:52.365059+0000 mgr.y (mgr.14144) 328 : audit [DBG] from='client.24802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:53.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:52.239235+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v297: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:53.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:52.365059+0000 mgr.y (mgr.14144) 328 : audit [DBG] from='client.24802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:53.728 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:24:53.728 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": {}, 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:24:53.729 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 17 2024-09-17T09:24:53.730 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:24:53.730 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:24:53.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:52.239235+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v297: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:53.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:52.365059+0000 mgr.y (mgr.14144) 328 : audit [DBG] from='client.24802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:54.105 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:24:54.105 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:24:54.105 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:24:54.106 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [], 2024-09-17T09:24:54.106 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "", 2024-09-17T09:24:54.106 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 image" 2024-09-17T09:24:54.106 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:24:54.502 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T09:24:54.502 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T09:24:54.502 INFO:teuthology.orchestra.run.smithi029.stdout: daemon iscsi.foo.smithi029.gellkh on smithi029 is in error state 2024-09-17T09:24:54.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:52.776279+0000 mgr.y (mgr.14144) 329 : audit [DBG] from='client.14889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:54.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:53.197624+0000 mgr.y (mgr.14144) 330 : audit [DBG] from='client.14895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:54.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:53.727934+0000 mon.c (mon.1) 26 : audit [DBG] from='client.? 172.21.15.29:0/2205804519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:24:54.503 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:52.776279+0000 mgr.y (mgr.14144) 329 : audit [DBG] from='client.14889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:54.503 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:53.197624+0000 mgr.y (mgr.14144) 330 : audit [DBG] from='client.14895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:54.503 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:53.727934+0000 mon.c (mon.1) 26 : audit [DBG] from='client.? 172.21.15.29:0/2205804519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:24:54.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:52.776279+0000 mgr.y (mgr.14144) 329 : audit [DBG] from='client.14889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:54.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:53.197624+0000 mgr.y (mgr.14144) 330 : audit [DBG] from='client.14895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:54.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:53.727934+0000 mon.c (mon.1) 26 : audit [DBG] from='client.? 172.21.15.29:0/2205804519' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:24:55.698 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:54.104888+0000 mgr.y (mgr.14144) 331 : audit [DBG] from='client.14907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:55.698 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:54.239723+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v298: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:55.698 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:24:54.501557+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? 172.21.15.29:0/2691943798' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:24:55.699 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:54.104888+0000 mgr.y (mgr.14144) 331 : audit [DBG] from='client.14907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:55.699 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:54.239723+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v298: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:55.699 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:24:54.501557+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? 172.21.15.29:0/2691943798' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:24:55.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:54.104888+0000 mgr.y (mgr.14144) 331 : audit [DBG] from='client.14907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:24:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:54.239723+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v298: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:24:54.501557+0000 mon.c (mon.1) 27 : audit [DBG] from='client.? 172.21.15.29:0/2691943798' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:24:57.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:56.240276+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v299: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:57.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:56.240276+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v299: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:57.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:56.240276+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v299: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:59.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:24:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:24:58.240738+0000 mgr.y (mgr.14144) 334 : cluster [DBG] pgmap v300: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:59.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:24:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:24:58.240738+0000 mgr.y (mgr.14144) 334 : cluster [DBG] pgmap v300: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:24:59.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:24:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:24:58.240738+0000 mgr.y (mgr.14144) 334 : cluster [DBG] pgmap v300: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:01.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:00.241242+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v301: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:01.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:00.241242+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v301: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:01.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:00.241242+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v301: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:03.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:02.241776+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v302: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:02.241776+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v302: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:03.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:02.241776+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v302: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:05.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:04.242241+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v303: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:05.861 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:04.683844+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:25:05.861 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:04.684040+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:25:05.865 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:04.242241+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v303: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:05.865 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:04.683844+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:25:05.865 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:04.684040+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:25:05.865 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:04.242241+0000 mgr.y (mgr.14144) 337 : cluster [DBG] pgmap v303: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:05.865 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:04.683844+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:25:05.865 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:04.684040+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:25:07.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:06.242860+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v304: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:07.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:06.242860+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v304: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:07.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:06.242860+0000 mgr.y (mgr.14144) 338 : cluster [DBG] pgmap v304: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:09.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:08.243344+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v305: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:09.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:08.243344+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v305: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:09.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:08.243344+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v305: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:11.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:10.244174+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v306: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:11.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:10.244174+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v306: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:11.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:10.244174+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v306: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:13.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:12.244741+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v307: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:13.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:12.244741+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v307: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:13.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:12.244741+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v307: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:15.558 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:14.245207+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v308: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:15.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:14.245207+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v308: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:15.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:14.245207+0000 mgr.y (mgr.14144) 342 : cluster [DBG] pgmap v308: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:17.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:16.246045+0000 mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v309: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:17.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:16.246045+0000 mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v309: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:17.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:16.246045+0000 mgr.y (mgr.14144) 343 : cluster [DBG] pgmap v309: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:19.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:17.607440+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:19.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:17.608031+0000 mgr.y (mgr.14144) 344 : cephadm [INF] Upgrade: Target is version 18.2.4-855-g19e2a2be (unknown) 2024-09-17T09:25:19.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:17.608099+0000 mgr.y (mgr.14144) 345 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9'] 2024-09-17T09:25:19.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:17.608925+0000 mon.a (mon.0) 634 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:25:19.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:17.609762+0000 mgr.y (mgr.14144) 346 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T09:25:19.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:17.812990+0000 mgr.y (mgr.14144) 347 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 on smithi103 2024-09-17T09:25:19.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:18.246680+0000 mgr.y (mgr.14144) 348 : cluster [DBG] pgmap v310: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:19.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:17.607440+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:19.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:17.608031+0000 mgr.y (mgr.14144) 344 : cephadm [INF] Upgrade: Target is version 18.2.4-855-g19e2a2be (unknown) 2024-09-17T09:25:19.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:17.608099+0000 mgr.y (mgr.14144) 345 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9'] 2024-09-17T09:25:19.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:17.608925+0000 mon.a (mon.0) 634 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:25:19.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:17.609762+0000 mgr.y (mgr.14144) 346 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T09:25:19.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:17.812990+0000 mgr.y (mgr.14144) 347 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 on smithi103 2024-09-17T09:25:19.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:18.246680+0000 mgr.y (mgr.14144) 348 : cluster [DBG] pgmap v310: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:19.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:17.607440+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:19.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:17.608031+0000 mgr.y (mgr.14144) 344 : cephadm [INF] Upgrade: Target is version 18.2.4-855-g19e2a2be (unknown) 2024-09-17T09:25:19.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:17.608099+0000 mgr.y (mgr.14144) 345 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:70b7810862ffedc21315a6c67a84878d4bbd16155279a41a670f6f6be7488da9'] 2024-09-17T09:25:19.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:17.608925+0000 mon.a (mon.0) 634 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:25:19.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:17.609762+0000 mgr.y (mgr.14144) 346 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T09:25:19.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:17.812990+0000 mgr.y (mgr.14144) 347 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 on smithi103 2024-09-17T09:25:19.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:18.246680+0000 mgr.y (mgr.14144) 348 : cluster [DBG] pgmap v310: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:21.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:20.247346+0000 mgr.y (mgr.14144) 349 : cluster [DBG] pgmap v311: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:21.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:20.247346+0000 mgr.y (mgr.14144) 349 : cluster [DBG] pgmap v311: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:21.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:20.247346+0000 mgr.y (mgr.14144) 349 : cluster [DBG] pgmap v311: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:23.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:22.248066+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v312: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:23.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:22.248066+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v312: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:23.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:22.248066+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v312: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:24.835 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:25:25.441 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID 2024-09-17T09:25:25.441 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 running (3m) 91s ago 3m *:9093 *:9094 0.20.0 0881eb8f169f a5a9a8d22f29 2024-09-17T09:25:25.441 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 running (3m) 2m ago 3m *:3000 6.7.4 557c83e11646 b74e78c8a6b5 2024-09-17T09:25:25.441 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 error 91s ago 2m - 2024-09-17T09:25:25.441 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 running (7m) 2m ago 7m *:8443 16.2.0 24ecd6d5f14c 1a97fa2150f2 2024-09-17T09:25:25.441 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 running (8m) 91s ago 8m *:9283 16.2.0 24ecd6d5f14c 9d130c67b36d 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (8m) 91s ago 9m - 16.2.0 24ecd6d5f14c f3476652dfdd 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (7m) 2m ago 7m - 16.2.0 24ecd6d5f14c 203d1f26d7b8 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (7m) 91s ago 7m - 16.2.0 24ecd6d5f14c aa05f3e5656a 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 running (4m) 91s ago 4m *:9100 0.18.1 e5a616e4b9cf 8202f7eb3e96 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 running (4m) 2m ago 4m *:9100 0.18.1 e5a616e4b9cf 0e6931583242 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (7m) 91s ago 7m - 16.2.0 24ecd6d5f14c 991731ce08b5 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (6m) 91s ago 6m - 16.2.0 24ecd6d5f14c d5bce63fadc8 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (6m) 91s ago 6m - 16.2.0 24ecd6d5f14c e9678d2f1685 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (6m) 91s ago 6m - 16.2.0 24ecd6d5f14c dcf7122a7d99 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (6m) 2m ago 6m - 16.2.0 24ecd6d5f14c 048c13025d32 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (5m) 2m ago 5m - 16.2.0 24ecd6d5f14c 10a64919fbc9 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (5m) 2m ago 5m - 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (5m) 2m ago 5m - 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:25:25.442 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 running (3m) 2m ago 4m *:9095 2.18.1 de242295e225 94209ce0b68f 2024-09-17T09:25:25.443 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 running (2m) 91s ago 2m *:8000 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:25:25.443 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 running (2m) 2m ago 2m *:8000 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:25:25.443 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 running (2m) 91s ago 2m *:80 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:25:25.443 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 running (2m) 2m ago 2m *:80 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:25:25.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:24.248698+0000 mgr.y (mgr.14144) 351 : cluster [DBG] pgmap v313: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:25.773 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:25:25.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:25:25.773 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2024-09-17T09:25:25.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:25:25.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:25:25.773 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2 2024-09-17T09:25:25.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:25:25.773 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:25:25.773 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T09:25:25.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:25:25.774 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": {}, 2024-09-17T09:25:25.774 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:25:25.774 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:25:25.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:25:25.774 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:25:25.774 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 17 2024-09-17T09:25:25.774 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:25:25.774 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:25:25.774 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:24.248698+0000 mgr.y (mgr.14144) 351 : cluster [DBG] pgmap v313: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:25.775 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:24.248698+0000 mgr.y (mgr.14144) 351 : cluster [DBG] pgmap v313: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:26.079 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:25:26.079 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:25:26.080 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:25:26.080 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [], 2024-09-17T09:25:26.080 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "0/23 ceph daemons upgraded", 2024-09-17T09:25:26.080 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 image on host smithi103" 2024-09-17T09:25:26.080 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:25:26.415 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T09:25:26.416 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T09:25:26.416 INFO:teuthology.orchestra.run.smithi029.stdout: daemon iscsi.foo.smithi029.gellkh on smithi029 is in error state 2024-09-17T09:25:26.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:24.822761+0000 mgr.y (mgr.14144) 352 : audit [DBG] from='client.14919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:26.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:25.133038+0000 mgr.y (mgr.14144) 353 : audit [DBG] from='client.24844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:26.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:25.432840+0000 mgr.y (mgr.14144) 354 : audit [DBG] from='client.24850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:26.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:25.772311+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.29:0/2015809770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:25:26.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:24.822761+0000 mgr.y (mgr.14144) 352 : audit [DBG] from='client.14919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:25.133038+0000 mgr.y (mgr.14144) 353 : audit [DBG] from='client.24844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:25.432840+0000 mgr.y (mgr.14144) 354 : audit [DBG] from='client.24850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:25.772311+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.29:0/2015809770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:25:26.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:24.822761+0000 mgr.y (mgr.14144) 352 : audit [DBG] from='client.14919 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:26.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:25.133038+0000 mgr.y (mgr.14144) 353 : audit [DBG] from='client.24844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:26.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:25.432840+0000 mgr.y (mgr.14144) 354 : audit [DBG] from='client.24850 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:25.772311+0000 mon.c (mon.1) 28 : audit [DBG] from='client.? 172.21.15.29:0/2015809770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:25:27.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:26.079023+0000 mgr.y (mgr.14144) 355 : audit [DBG] from='client.24862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:27.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:26.249447+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v314: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:27.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:26.414829+0000 mon.a (mon.0) 635 : audit [DBG] from='client.? 172.21.15.29:0/3338280634' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:25:27.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:26.079023+0000 mgr.y (mgr.14144) 355 : audit [DBG] from='client.24862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:27.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:26.249447+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v314: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:27.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:26.414829+0000 mon.a (mon.0) 635 : audit [DBG] from='client.? 172.21.15.29:0/3338280634' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:25:27.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:26.079023+0000 mgr.y (mgr.14144) 355 : audit [DBG] from='client.24862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:27.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:26.249447+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v314: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:27.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:26.414829+0000 mon.a (mon.0) 635 : audit [DBG] from='client.? 172.21.15.29:0/3338280634' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:25:29.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:28.250145+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v315: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:29.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:28.250145+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v315: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:28.250145+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v315: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:31.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:30.250938+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v316: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:31.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:30.250938+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v316: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:31.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:30.250938+0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v316: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:33.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:32.251769+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v317: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:33.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:32.251769+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v317: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:33.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:32.251769+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v317: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:35.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:34.252424+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v318: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:35.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:34.252424+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v318: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:35.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:34.252424+0000 mgr.y (mgr.14144) 360 : cluster [DBG] pgmap v318: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:37.593 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:36.253194+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v319: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:37.593 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:36.253194+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v319: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:37.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:36.253194+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v319: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:39.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:38.253815+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v320: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:39.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:38.253815+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v320: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:39.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:38.253815+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v320: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:41.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:40.254376+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v321: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:41.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:40.254376+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v321: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:41.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:40.254376+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v321: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:43.729 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:42.255138+0000 mgr.y (mgr.14144) 364 : cluster [DBG] pgmap v322: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:43.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:42.255138+0000 mgr.y (mgr.14144) 364 : cluster [DBG] pgmap v322: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:43.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:42.255138+0000 mgr.y (mgr.14144) 364 : cluster [DBG] pgmap v322: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:44.094419+0000 mgr.y (mgr.14144) 365 : cephadm [INF] Upgrade: Updating mgr.x 2024-09-17T09:25:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:44.101127+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:44.102092+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:25:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:44.102983+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:25:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:44.103763+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:25:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:44.104546+0000 mgr.y (mgr.14144) 366 : cephadm [INF] Deploying daemon mgr.x on smithi103 2024-09-17T09:25:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:44.255928+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v323: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:45.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:44.094419+0000 mgr.y (mgr.14144) 365 : cephadm [INF] Upgrade: Updating mgr.x 2024-09-17T09:25:45.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:44.101127+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:45.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:44.102092+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:25:45.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:44.102983+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:25:45.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:44.103763+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:25:45.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:44.104546+0000 mgr.y (mgr.14144) 366 : cephadm [INF] Deploying daemon mgr.x on smithi103 2024-09-17T09:25:45.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:44.255928+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v323: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:45.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:44.094419+0000 mgr.y (mgr.14144) 365 : cephadm [INF] Upgrade: Updating mgr.x 2024-09-17T09:25:45.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:44.101127+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:45.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:44.102092+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:25:45.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:44.102983+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:25:45.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:44.103763+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:25:45.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:44.104546+0000 mgr.y (mgr.14144) 366 : cephadm [INF] Deploying daemon mgr.x on smithi103 2024-09-17T09:25:45.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:44.255928+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v323: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:45.688 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:45 smithi103 systemd[1]: Stopping Ceph mgr.x for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:25:46.004 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:45 smithi103 podman[40800]: 2024-09-17 09:25:45.692134965 +0000 UTC m=+0.106155152 container died 1a97fa2150f2908fc5d8643f9e897435075aa45292ab3abca9bd0cf0d78cff1c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x, org.label-schema.vendor=CentOS, org.label-schema.build-date=20201204, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, RELEASE=HEAD, ceph=True, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=-16.2.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-17T09:25:46.004 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 podman[40800]: 2024-09-17 09:25:46.003932444 +0000 UTC m=+0.417952633 container cleanup 1a97fa2150f2908fc5d8643f9e897435075aa45292ab3abca9bd0cf0d78cff1c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x, ceph=True, maintainer=Dimitri Savineau , GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.build-date=20201204, RELEASE=HEAD, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS) 2024-09-17T09:25:46.255 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 podman[40800]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x 2024-09-17T09:25:46.255 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 podman[40812]: 2024-09-17 09:25:46.197029874 +0000 UTC m=+0.502327049 container remove 1a97fa2150f2908fc5d8643f9e897435075aa45292ab3abca9bd0cf0d78cff1c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x, GIT_CLEAN=True, ceph=True, maintainer=Dimitri Savineau , GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) 2024-09-17T09:25:46.256 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:25:46.256 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-17T09:25:46.256 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 systemd[1]: Stopped Ceph mgr.x for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:25:46.706 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 systemd[1]: Starting Ceph mgr.x for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:25:46.706 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 podman[40891]: 2024-09-17 09:25:46.491131804 +0000 UTC m=+0.015882372 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:25:46.706 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 podman[40891]: 2024-09-17 09:25:46.598354704 +0000 UTC m=+0.123105304 container create d35594bdef289e64437c1e6b911c9ce4c90ee72bb7ea0bbea38a117e83122c94 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-17T09:25:46.978 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 podman[40891]: 2024-09-17 09:25:46.850639424 +0000 UTC m=+0.375389986 container init d35594bdef289e64437c1e6b911c9ce4c90ee72bb7ea0bbea38a117e83122c94 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:25:46.979 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 podman[40891]: 2024-09-17 09:25:46.853672958 +0000 UTC m=+0.378423520 container start d35594bdef289e64437c1e6b911c9ce4c90ee72bb7ea0bbea38a117e83122c94 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_CLEAN=True) 2024-09-17T09:25:46.979 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:46.876+0000 7fae0e1d6640 1 -- 172.21.15.103:0/3333723311 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5609ee544d00 con 0x5609ee519000 2024-09-17T09:25:46.979 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 bash[40891]: d35594bdef289e64437c1e6b911c9ce4c90ee72bb7ea0bbea38a117e83122c94 2024-09-17T09:25:46.979 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 systemd[1]: Started Ceph mgr.x for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:25:46.979 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:46.976+0000 7fae10a3f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:25:47.255 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:47.036+0000 7fae10a3f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:25:47.256 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:47.201+0000 7fae10a3f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:25:47.576 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:46.256681+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v324: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:47.576 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:46.997171+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:47.577 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:47.274+0000 7fae10a3f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:25:47.577 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:47.424+0000 7fae10a3f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:25:47.577 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:47.574+0000 7fae10a3f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:25:47.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:46.256681+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v324: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:47.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:46.997171+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:47.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:46.256681+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v324: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:47.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:46.997171+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:48.128 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:47.903+0000 7fae10a3f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:25:48.128 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:47.965+0000 7fae10a3f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:25:48.416 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:48.151+0000 7fae10a3f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:25:49.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:48.257387+0000 mgr.y (mgr.14144) 369 : cluster [DBG] pgmap v325: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:49.256 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:49.084+0000 7fae10a3f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:25:49.256 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:49.157+0000 7fae10a3f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:25:49.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:48.257387+0000 mgr.y (mgr.14144) 369 : cluster [DBG] pgmap v325: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:49.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:48.257387+0000 mgr.y (mgr.14144) 369 : cluster [DBG] pgmap v325: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:50.191 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:49.921+0000 7fae10a3f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:25:50.192 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:50.042+0000 7fae10a3f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:25:50.481 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:50.189+0000 7fae10a3f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:25:50.481 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:50.253+0000 7fae10a3f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:25:50.481 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:50.408+0000 7fae10a3f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:25:50.481 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:50.479+0000 7fae10a3f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:25:50.755 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:50.539+0000 7fae10a3f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:25:50.755 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:50.699+0000 7fae10a3f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:25:51.038 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:50.764+0000 7fae10a3f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:25:51.038 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:50.832+0000 7fae10a3f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:25:51.038 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:50.917+0000 7fae10a3f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:25:51.038 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:50.976+0000 7fae10a3f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:25:51.038 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:51.035+0000 7fae10a3f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:25:51.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:50.258182+0000 mgr.y (mgr.14144) 370 : cluster [DBG] pgmap v326: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:51.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:51.219582+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:51.797 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:50.258182+0000 mgr.y (mgr.14144) 370 : cluster [DBG] pgmap v326: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:51.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:51.219582+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:51.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:50.258182+0000 mgr.y (mgr.14144) 370 : cluster [DBG] pgmap v326: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:51.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:51.219582+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:52.255 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:51.842+0000 7fae10a3f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:25:52.702 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:51.437545+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:52.702 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:51.441968+0000 mon.a (mon.0) 643 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:25:52.702 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:51.443017+0000 mgr.y (mgr.14144) 371 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T09:25:52.702 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:51.445380+0000 mgr.y (mgr.14144) 372 : cephadm [INF] Failing over to other MGR 2024-09-17T09:25:52.702 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:51.445860+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T09:25:52.702 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:51.450659+0000 mon.a (mon.0) 645 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-17T09:25:52.702 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:51.845583+0000 mon.a (mon.0) 646 : cluster [INF] Active manager daemon x restarted 2024-09-17T09:25:52.702 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:51.845946+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:25:52.702 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:51.846294+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:25:52.702 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:51.846603+0000 mon.a (mon.0) 647 : cluster [INF] Activating manager daemon x 2024-09-17T09:25:52.703 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:51.846973+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.? 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:25:52.703 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:51.850913+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-17T09:25:52.703 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:51.851543+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.? 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:25:52.751 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: ignoring --setuser ceph since I am not root 2024-09-17T09:25:52.751 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: ignoring --setgroup ceph since I am not root 2024-09-17T09:25:52.751 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y[23832]: debug 2024-09-17T09:25:52.489+0000 7fe9d7de3700 1 -- 172.21.15.29:0/4123977187 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561d969004e0 con 0x561d968d7400 2024-09-17T09:25:52.751 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:51.437545+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:52.751 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:51.441968+0000 mon.a (mon.0) 643 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:25:52.751 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:51.443017+0000 mgr.y (mgr.14144) 371 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T09:25:52.751 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:51.445380+0000 mgr.y (mgr.14144) 372 : cephadm [INF] Failing over to other MGR 2024-09-17T09:25:52.751 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:51.445860+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T09:25:52.752 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:51.450659+0000 mon.a (mon.0) 645 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-17T09:25:52.752 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:51.845583+0000 mon.a (mon.0) 646 : cluster [INF] Active manager daemon x restarted 2024-09-17T09:25:52.752 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:51.845946+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:25:52.752 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:51.846294+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:25:52.752 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:51.846603+0000 mon.a (mon.0) 647 : cluster [INF] Activating manager daemon x 2024-09-17T09:25:52.752 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:51.846973+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.? 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:25:52.752 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:51.850913+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-17T09:25:52.752 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:51.851543+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.? 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:25:52.752 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:51.437545+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' 2024-09-17T09:25:52.752 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:51.441968+0000 mon.a (mon.0) 643 : audit [DBG] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:25:52.752 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:51.443017+0000 mgr.y (mgr.14144) 371 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T09:25:52.752 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:51.445380+0000 mgr.y (mgr.14144) 372 : cephadm [INF] Failing over to other MGR 2024-09-17T09:25:52.753 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:51.445860+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T09:25:52.753 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:51.450659+0000 mon.a (mon.0) 645 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-17T09:25:52.753 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:51.845583+0000 mon.a (mon.0) 646 : cluster [INF] Active manager daemon x restarted 2024-09-17T09:25:52.753 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:51.845946+0000 mon.b (mon.2) 24 : audit [DBG] from='mgr.? 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:25:52.753 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:51.846294+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:25:52.753 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:51.846603+0000 mon.a (mon.0) 647 : cluster [INF] Activating manager daemon x 2024-09-17T09:25:52.753 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:51.846973+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.? 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:25:52.753 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:51.850913+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-17T09:25:52.753 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:51.851543+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.? 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:25:53.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.443768+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T09:25:53.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:52.443882+0000 mon.a (mon.0) 650 : cluster [DBG] mgrmap e16: x(active, starting, since 0.597612s) 2024-09-17T09:25:53.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.444092+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:25:53.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.444397+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:25:53.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.444579+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:25:53.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.445447+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:25:53.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.445556+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:25:53.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.445658+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:25:53.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.445802+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:25:53.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.445944+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:25:53.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.449197+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:25:53.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.449293+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:25:53.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.449377+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:25:53.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.449489+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:25:53.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.451156+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:25:53.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.451313+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:25:53.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.451776+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:25:53.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:52.531745+0000 mon.a (mon.0) 651 : cluster [INF] Manager daemon x is now available 2024-09-17T09:25:53.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:52.541590+0000 mgr.x (mgr.34775) 1 : cephadm [INF] inventory: adjusted host smithi029 addr 'smithi029' -> '172.21.15.29' 2024-09-17T09:25:53.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.545141+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:53.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.553853+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:53.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:52.554287+0000 mgr.x (mgr.34775) 2 : cephadm [INF] Queued rgw.foo for migration 2024-09-17T09:25:53.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.557263+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:53.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:52.557622+0000 mgr.x (mgr.34775) 3 : cephadm [INF] Queued rgw.smpl for migration 2024-09-17T09:25:53.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.579186+0000 mon.b (mon.2) 43 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:25:53.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.579550+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:25:53.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.588690+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:53.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.589424+0000 mon.b (mon.2) 44 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T09:25:53.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:52.589740+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T09:25:53.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.443768+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T09:25:53.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:52.443882+0000 mon.a (mon.0) 650 : cluster [DBG] mgrmap e16: x(active, starting, since 0.597612s) 2024-09-17T09:25:53.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.444092+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:25:53.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.444397+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:25:53.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.444579+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:25:53.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.445447+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:25:53.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.445556+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:25:53.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.445658+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:25:53.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.445802+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:25:53.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.445944+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:25:53.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.449197+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:25:53.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.449293+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:25:53.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.449377+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:25:53.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.449489+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:25:53.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.451156+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:25:53.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.443768+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14144 172.21.15.29:0/435578566' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T09:25:53.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:52.443882+0000 mon.a (mon.0) 650 : cluster [DBG] mgrmap e16: x(active, starting, since 0.597612s) 2024-09-17T09:25:53.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.444092+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:25:53.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.444397+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:25:53.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.444579+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:25:53.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.445447+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:25:53.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.445556+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:25:53.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.445658+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:25:53.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.445802+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:25:53.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.445944+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:25:53.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.449197+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:25:53.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.449293+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:25:53.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.449377+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:25:53.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.449489+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:25:53.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.451156+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:25:53.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.451313+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:25:53.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.451776+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:25:53.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:52.531745+0000 mon.a (mon.0) 651 : cluster [INF] Manager daemon x is now available 2024-09-17T09:25:53.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:52.541590+0000 mgr.x (mgr.34775) 1 : cephadm [INF] inventory: adjusted host smithi029 addr 'smithi029' -> '172.21.15.29' 2024-09-17T09:25:53.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.545141+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:53.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.553853+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:53.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:52.554287+0000 mgr.x (mgr.34775) 2 : cephadm [INF] Queued rgw.foo for migration 2024-09-17T09:25:53.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.557263+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:53.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:52.557622+0000 mgr.x (mgr.34775) 3 : cephadm [INF] Queued rgw.smpl for migration 2024-09-17T09:25:53.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.579186+0000 mon.b (mon.2) 43 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:25:53.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.579550+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:25:53.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.588690+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:53.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.589424+0000 mon.b (mon.2) 44 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T09:25:53.802 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:52.589740+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T09:25:53.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.451313+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:25:53.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.451776+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:25:53.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:52.531745+0000 mon.a (mon.0) 651 : cluster [INF] Manager daemon x is now available 2024-09-17T09:25:53.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:52.541590+0000 mgr.x (mgr.34775) 1 : cephadm [INF] inventory: adjusted host smithi029 addr 'smithi029' -> '172.21.15.29' 2024-09-17T09:25:53.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.545141+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:53.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.553853+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:53.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:52.554287+0000 mgr.x (mgr.34775) 2 : cephadm [INF] Queued rgw.foo for migration 2024-09-17T09:25:53.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.557263+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:53.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:52.557622+0000 mgr.x (mgr.34775) 3 : cephadm [INF] Queued rgw.smpl for migration 2024-09-17T09:25:53.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.579186+0000 mon.b (mon.2) 43 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:25:53.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.579550+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:25:53.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.588690+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:53.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.589424+0000 mon.b (mon.2) 44 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T09:25:53.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:52.589740+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T09:25:54.740 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:53.048511+0000 mgr.x (mgr.34775) 4 : cephadm [INF] Deploying cephadm binary to smithi103 2024-09-17T09:25:54.741 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:53.399623+0000 mgr.x (mgr.34775) 5 : cephadm [INF] Deploying cephadm binary to smithi029 2024-09-17T09:25:54.741 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:53.453470+0000 mon.a (mon.0) 658 : cluster [DBG] mgrmap e17: x(active, since 1.60721s) 2024-09-17T09:25:54.741 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:53.468725+0000 mgr.x (mgr.34775) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:54.741 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:53.519868+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:54.741 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:53.524415+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:54.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:53.048511+0000 mgr.x (mgr.34775) 4 : cephadm [INF] Deploying cephadm binary to smithi103 2024-09-17T09:25:54.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:53.399623+0000 mgr.x (mgr.34775) 5 : cephadm [INF] Deploying cephadm binary to smithi029 2024-09-17T09:25:54.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:53.453470+0000 mon.a (mon.0) 658 : cluster [DBG] mgrmap e17: x(active, since 1.60721s) 2024-09-17T09:25:54.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:53.468725+0000 mgr.x (mgr.34775) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:54.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:53.519868+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:54.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:53.524415+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:54.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:53.048511+0000 mgr.x (mgr.34775) 4 : cephadm [INF] Deploying cephadm binary to smithi103 2024-09-17T09:25:54.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:53.399623+0000 mgr.x (mgr.34775) 5 : cephadm [INF] Deploying cephadm binary to smithi029 2024-09-17T09:25:54.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:53.453470+0000 mon.a (mon.0) 658 : cluster [DBG] mgrmap e17: x(active, since 1.60721s) 2024-09-17T09:25:54.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:53.468725+0000 mgr.x (mgr.34775) 6 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:54.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:53.519868+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:54.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:53.524415+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:55.722 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:54.448539+0000 mgr.x (mgr.34775) 7 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:55.722 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:54.457483+0000 mon.a (mon.0) 661 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T09:25:55.722 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:54.457509+0000 mon.a (mon.0) 662 : cluster [INF] Cluster is now healthy 2024-09-17T09:25:55.722 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:54.527440+0000 mon.a (mon.0) 663 : cluster [DBG] mgrmap e18: x(active, since 2s) 2024-09-17T09:25:55.723 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:54.569968+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:25:55.723 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:54.570485+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:25:55.723 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:54.448539+0000 mgr.x (mgr.34775) 7 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:55.723 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:54.457483+0000 mon.a (mon.0) 661 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T09:25:55.723 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:54.457509+0000 mon.a (mon.0) 662 : cluster [INF] Cluster is now healthy 2024-09-17T09:25:55.723 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:54.527440+0000 mon.a (mon.0) 663 : cluster [DBG] mgrmap e18: x(active, since 2s) 2024-09-17T09:25:55.723 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:54.569968+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:25:55.723 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:54.570485+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:25:55.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:54.448539+0000 mgr.x (mgr.34775) 7 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:54.457483+0000 mon.a (mon.0) 661 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T09:25:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:54.457509+0000 mon.a (mon.0) 662 : cluster [INF] Cluster is now healthy 2024-09-17T09:25:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:54.527440+0000 mon.a (mon.0) 663 : cluster [DBG] mgrmap e18: x(active, since 2s) 2024-09-17T09:25:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:54.569968+0000 mon.b (mon.2) 45 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:25:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:54.570485+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:25:55.756 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:25:55.577+0000 7fad7664d640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:09:25:55] ENGINE Error in HTTPServer.serve 2024-09-17T09:25:55.756 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: Traceback (most recent call last): 2024-09-17T09:25:55.756 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:25:55.756 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: self._connections.run(self.expiration_interval) 2024-09-17T09:25:55.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:25:55.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: self._run(expiration_interval) 2024-09-17T09:25:55.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:25:55.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:25:55.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:25:55.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:25:55.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:25:55.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: s = self.context.wrap_socket( 2024-09-17T09:25:55.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:25:55.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: return self.sslsocket_class._create( 2024-09-17T09:25:55.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:25:55.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: self.do_handshake() 2024-09-17T09:25:55.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:25:55.758 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: self._sslobj.do_handshake() 2024-09-17T09:25:55.758 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:25:55.758 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:25:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: 2024-09-17T09:25:56.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:55.459064+0000 mgr.x (mgr.34775) 8 : cephadm [INF] [17/Sep/2024:09:25:55] ENGINE Bus STARTING 2024-09-17T09:25:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:55.527611+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T09:25:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:55.527696+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-17T09:25:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:55.533375+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:25:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:55.533845+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:25:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:55.576017+0000 mgr.x (mgr.34775) 9 : cephadm [INF] [17/Sep/2024:09:25:55] ENGINE Serving on https://172.21.15.103:7150 2024-09-17T09:25:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:55.578504+0000 mgr.x (mgr.34775) 10 : cephadm [ERR] [17/Sep/2024:09:25:55] ENGINE Error in HTTPServer.serve 2024-09-17T09:25:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Traceback (most recent call last): 2024-09-17T09:25:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:25:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: self._connections.run(self.expiration_interval) 2024-09-17T09:25:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:25:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: self._run(expiration_interval) 2024-09-17T09:25:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:25:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: s = self.context.wrap_socket( 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: return self.sslsocket_class._create( 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: self.do_handshake() 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: self._sslobj.do_handshake() 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:55.677209+0000 mgr.x (mgr.34775) 11 : cephadm [INF] [17/Sep/2024:09:25:55] ENGINE Serving on http://172.21.15.103:8765 2024-09-17T09:25:56.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:25:55.677293+0000 mgr.x (mgr.34775) 12 : cephadm [INF] [17/Sep/2024:09:25:55] ENGINE Bus STARTED 2024-09-17T09:25:56.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:55.459064+0000 mgr.x (mgr.34775) 8 : cephadm [INF] [17/Sep/2024:09:25:55] ENGINE Bus STARTING 2024-09-17T09:25:56.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:55.527611+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T09:25:56.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:55.527696+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-17T09:25:56.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:55.533375+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:25:56.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:55.533845+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:25:56.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:55.576017+0000 mgr.x (mgr.34775) 9 : cephadm [INF] [17/Sep/2024:09:25:55] ENGINE Serving on https://172.21.15.103:7150 2024-09-17T09:25:56.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:55.578504+0000 mgr.x (mgr.34775) 10 : cephadm [ERR] [17/Sep/2024:09:25:55] ENGINE Error in HTTPServer.serve 2024-09-17T09:25:56.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Traceback (most recent call last): 2024-09-17T09:25:56.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:25:56.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: self._connections.run(self.expiration_interval) 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: self._run(expiration_interval) 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: s = self.context.wrap_socket( 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: return self.sslsocket_class._create( 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: self.do_handshake() 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:25:56.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: self._sslobj.do_handshake() 2024-09-17T09:25:56.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:25:56.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:25:56.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:55.677209+0000 mgr.x (mgr.34775) 11 : cephadm [INF] [17/Sep/2024:09:25:55] ENGINE Serving on http://172.21.15.103:8765 2024-09-17T09:25:56.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:25:55.677293+0000 mgr.x (mgr.34775) 12 : cephadm [INF] [17/Sep/2024:09:25:55] ENGINE Bus STARTED 2024-09-17T09:25:56.825 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:25:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:55.459064+0000 mgr.x (mgr.34775) 8 : cephadm [INF] [17/Sep/2024:09:25:55] ENGINE Bus STARTING 2024-09-17T09:25:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:55.527611+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd='[{"prefix": "osd pool rename", "format": "json", "srcpool": "device_health_metrics", "destpool": ".mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T09:25:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:55.527696+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-17T09:25:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:55.533375+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:25:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:55.533845+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T09:25:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:55.576017+0000 mgr.x (mgr.34775) 9 : cephadm [INF] [17/Sep/2024:09:25:55] ENGINE Serving on https://172.21.15.103:7150 2024-09-17T09:25:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:55.578504+0000 mgr.x (mgr.34775) 10 : cephadm [ERR] [17/Sep/2024:09:25:55] ENGINE Error in HTTPServer.serve 2024-09-17T09:25:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Traceback (most recent call last): 2024-09-17T09:25:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:25:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: self._connections.run(self.expiration_interval) 2024-09-17T09:25:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:25:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: self._run(expiration_interval) 2024-09-17T09:25:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:25:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:25:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:25:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:25:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:25:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: s = self.context.wrap_socket( 2024-09-17T09:25:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:25:57.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: return self.sslsocket_class._create( 2024-09-17T09:25:57.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:25:57.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: self.do_handshake() 2024-09-17T09:25:57.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:25:57.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: self._sslobj.do_handshake() 2024-09-17T09:25:57.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:25:57.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:25:57.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:55.677209+0000 mgr.x (mgr.34775) 11 : cephadm [INF] [17/Sep/2024:09:25:55] ENGINE Serving on http://172.21.15.103:8765 2024-09-17T09:25:57.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:25:55.677293+0000 mgr.x (mgr.34775) 12 : cephadm [INF] [17/Sep/2024:09:25:55] ENGINE Bus STARTED 2024-09-17T09:25:57.542 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:25:57.542 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (3m) 2m ago 4m 19.0M - 0.20.0 0881eb8f169f a5a9a8d22f29 2024-09-17T09:25:57.542 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (3m) 6s ago 3m 35.3M - 6.7.4 557c83e11646 b74e78c8a6b5 2024-09-17T09:25:57.543 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 error 2m ago 3m - - 2024-09-17T09:25:57.543 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443 running (11s) 6s ago 8m 57.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 d35594bdef28 2024-09-17T09:25:57.543 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:9283 running (9m) 2m ago 9m 405M - 16.2.0 24ecd6d5f14c 9d130c67b36d 2024-09-17T09:25:57.543 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (9m) 2m ago 9m 46.5M 2048M 16.2.0 24ecd6d5f14c f3476652dfdd 2024-09-17T09:25:57.543 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (8m) 6s ago 8m 46.3M 2048M 16.2.0 24ecd6d5f14c 203d1f26d7b8 2024-09-17T09:25:57.543 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (8m) 2m ago 8m 38.5M 2048M 16.2.0 24ecd6d5f14c aa05f3e5656a 2024-09-17T09:25:57.543 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (4m) 2m ago 4m 14.9M - 0.18.1 e5a616e4b9cf 8202f7eb3e96 2024-09-17T09:25:57.543 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (4m) 6s ago 4m 16.9M - 0.18.1 e5a616e4b9cf 0e6931583242 2024-09-17T09:25:57.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (7m) 2m ago 7m 44.0M 4096M 16.2.0 24ecd6d5f14c 991731ce08b5 2024-09-17T09:25:57.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (7m) 2m ago 7m 43.8M 4096M 16.2.0 24ecd6d5f14c d5bce63fadc8 2024-09-17T09:25:57.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (7m) 2m ago 7m 41.0M 4096M 16.2.0 24ecd6d5f14c e9678d2f1685 2024-09-17T09:25:57.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (6m) 2m ago 6m 47.3M 4096M 16.2.0 24ecd6d5f14c dcf7122a7d99 2024-09-17T09:25:57.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (6m) 6s ago 6m 47.5M 4096M 16.2.0 24ecd6d5f14c 048c13025d32 2024-09-17T09:25:57.544 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (6m) 6s ago 6m 45.5M 4096M 16.2.0 24ecd6d5f14c 10a64919fbc9 2024-09-17T09:25:57.544 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (5m) 6s ago 5m 42.7M 4096M 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:25:57.544 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (5m) 6s ago 5m 48.2M 4096M 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:25:57.544 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 running (3m) 6s ago 5m 27.0M - 2.18.1 de242295e225 94209ce0b68f 2024-09-17T09:25:57.544 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (3m) 2m ago 3m 37.8M - 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:25:57.544 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (3m) 6s ago 3m 38.4M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:25:57.544 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (3m) 2m ago 3m 35.2M - 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:25:57.544 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (3m) 6s ago 3m 35.7M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:25:57.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:56.448931+0000 mgr.x (mgr.34775) 13 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:57.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:56.535192+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T09:25:57.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:56.535288+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-17T09:25:57.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:56.796699+0000 mgr.x (mgr.34775) 14 : audit [DBG] from='client.24901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:57.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:56.448931+0000 mgr.x (mgr.34775) 13 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:57.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:56.535192+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T09:25:57.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:56.535288+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-17T09:25:57.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:56.796699+0000 mgr.x (mgr.34775) 14 : audit [DBG] from='client.24901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:57.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:56.448931+0000 mgr.x (mgr.34775) 13 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:57.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:56.535192+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T09:25:57.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:56.535288+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-17T09:25:57.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:56.796699+0000 mgr.x (mgr.34775) 14 : audit [DBG] from='client.24901 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:58.017 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:25:58.017 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:25:58.017 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2024-09-17T09:25:58.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:25:58.017 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:25:58.017 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T09:25:58.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:25:58.017 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:25:58.017 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T09:25:58.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:25:58.018 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": {}, 2024-09-17T09:25:58.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:25:58.018 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:25:58.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:25:58.018 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:25:58.018 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 15, 2024-09-17T09:25:58.018 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T09:25:58.018 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:25:58.018 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:25:58.357 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:25:58.358 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:25:58.358 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:25:58.358 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:25:58.358 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [], 2024-09-17T09:25:58.358 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "1/23 daemons upgraded", 2024-09-17T09:25:58.358 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "", 2024-09-17T09:25:58.358 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:25:58.358 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:25:58.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:57.182148+0000 mgr.x (mgr.34775) 15 : audit [DBG] from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:58.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:57.531175+0000 mgr.x (mgr.34775) 16 : audit [DBG] from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:58.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:57.562228+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:58.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:57.784132+0000 mon.a (mon.0) 671 : cluster [DBG] Standby manager daemon y started 2024-09-17T09:25:58.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:57.786447+0000 mon.a (mon.0) 672 : audit [DBG] from='mgr.? 172.21.15.29:0/3809683156' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:25:58.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:57.786989+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.? 172.21.15.29:0/3809683156' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:25:58.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:57.788069+0000 mon.a (mon.0) 674 : audit [DBG] from='mgr.? 172.21.15.29:0/3809683156' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T09:25:58.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:57.788384+0000 mon.a (mon.0) 675 : audit [DBG] from='mgr.? 172.21.15.29:0/3809683156' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:25:58.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:58.016306+0000 mon.a (mon.0) 676 : audit [DBG] from='client.? 172.21.15.29:0/1775733933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:25:58.760 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_OK 2024-09-17T09:25:58.760 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:57.182148+0000 mgr.x (mgr.34775) 15 : audit [DBG] from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:58.760 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:57.531175+0000 mgr.x (mgr.34775) 16 : audit [DBG] from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:58.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:57.562228+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:58.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:57.784132+0000 mon.a (mon.0) 671 : cluster [DBG] Standby manager daemon y started 2024-09-17T09:25:58.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:57.786447+0000 mon.a (mon.0) 672 : audit [DBG] from='mgr.? 172.21.15.29:0/3809683156' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:25:58.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:57.786989+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.? 172.21.15.29:0/3809683156' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:25:58.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:57.788069+0000 mon.a (mon.0) 674 : audit [DBG] from='mgr.? 172.21.15.29:0/3809683156' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T09:25:58.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:57.788384+0000 mon.a (mon.0) 675 : audit [DBG] from='mgr.? 172.21.15.29:0/3809683156' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:25:58.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:58.016306+0000 mon.a (mon.0) 676 : audit [DBG] from='client.? 172.21.15.29:0/1775733933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:25:58.761 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:57.182148+0000 mgr.x (mgr.34775) 15 : audit [DBG] from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:58.762 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:57.531175+0000 mgr.x (mgr.34775) 16 : audit [DBG] from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:58.762 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:57.562228+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:58.762 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:57.784132+0000 mon.a (mon.0) 671 : cluster [DBG] Standby manager daemon y started 2024-09-17T09:25:58.762 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:57.786447+0000 mon.a (mon.0) 672 : audit [DBG] from='mgr.? 172.21.15.29:0/3809683156' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:25:58.762 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:57.786989+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.? 172.21.15.29:0/3809683156' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:25:58.762 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:57.788069+0000 mon.a (mon.0) 674 : audit [DBG] from='mgr.? 172.21.15.29:0/3809683156' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T09:25:58.762 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:57.788384+0000 mon.a (mon.0) 675 : audit [DBG] from='mgr.? 172.21.15.29:0/3809683156' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:25:58.762 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:58.016306+0000 mon.a (mon.0) 676 : audit [DBG] from='client.? 172.21.15.29:0/1775733933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:25:59.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:58.356809+0000 mgr.x (mgr.34775) 17 : audit [DBG] from='client.24931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:59.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:58.449285+0000 mgr.x (mgr.34775) 18 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:59.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:25:58.572108+0000 mon.a (mon.0) 677 : cluster [DBG] mgrmap e19: x(active, since 6s), standbys: y 2024-09-17T09:25:59.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:58.573150+0000 mon.b (mon.2) 47 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:25:59.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:58.760147+0000 mon.a (mon.0) 678 : audit [DBG] from='client.? 172.21.15.29:0/3530924005' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:25:59.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:58.910086+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:59.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:58.915895+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:59.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:59.408918+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:59.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:25:59.414108+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:59.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:58.356809+0000 mgr.x (mgr.34775) 17 : audit [DBG] from='client.24931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:25:59.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:58.449285+0000 mgr.x (mgr.34775) 18 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:25:59.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:25:58.572108+0000 mon.a (mon.0) 677 : cluster [DBG] mgrmap e19: x(active, since 6s), standbys: y 2024-09-17T09:25:59.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:58.573150+0000 mon.b (mon.2) 47 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:25:59.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:58.760147+0000 mon.a (mon.0) 678 : audit [DBG] from='client.? 172.21.15.29:0/3530924005' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:25:59.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:58.910086+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:59.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:58.915895+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:59.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:59.408918+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:25:59.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:25:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:25:59.414108+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:58.356809+0000 mgr.x (mgr.34775) 17 : audit [DBG] from='client.24931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:26:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:58.449285+0000 mgr.x (mgr.34775) 18 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 27 KiB data, 89 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:25:58.572108+0000 mon.a (mon.0) 677 : cluster [DBG] mgrmap e19: x(active, since 6s), standbys: y 2024-09-17T09:26:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:58.573150+0000 mon.b (mon.2) 47 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:26:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:58.760147+0000 mon.a (mon.0) 678 : audit [DBG] from='client.? 172.21.15.29:0/3530924005' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:26:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:58.910086+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:58.915895+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:59.408918+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:00.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:25:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:25:59.414108+0000 mon.a (mon.0) 682 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:02.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:00.449943+0000 mgr.x (mgr.34775) 19 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 234 KiB/s wr, 31 op/s 2024-09-17T09:26:02.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:00.802760+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:02.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:00.449943+0000 mgr.x (mgr.34775) 19 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 234 KiB/s wr, 31 op/s 2024-09-17T09:26:02.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:00.802760+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:02.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:00.449943+0000 mgr.x (mgr.34775) 19 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 234 KiB/s wr, 31 op/s 2024-09-17T09:26:02.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:00.802760+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:04.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:02.320395+0000 mon.a (mon.0) 684 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:26:04.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:02.320623+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:26:04.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:02.320782+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:26:04.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:02.320911+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:26:04.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:02.450308+0000 mgr.x (mgr.34775) 20 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 204 KiB/s wr, 27 op/s 2024-09-17T09:26:04.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:03.256639+0000 mon.a (mon.0) 685 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:26:04.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:03.258906+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:26:04.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:03.259120+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:26:04.490 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:02.320395+0000 mon.a (mon.0) 684 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:26:04.490 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:02.320623+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:26:04.491 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:02.320782+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:26:04.491 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:02.320911+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:26:04.491 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:02.450308+0000 mgr.x (mgr.34775) 20 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 204 KiB/s wr, 27 op/s 2024-09-17T09:26:04.491 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:03.256639+0000 mon.a (mon.0) 685 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:26:04.491 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:03.258906+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:26:04.491 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:03.259120+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:26:05.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:02.320395+0000 mon.a (mon.0) 684 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:26:05.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:02.320623+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:26:05.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:02.320782+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:26:05.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:02.320911+0000 mon.b (mon.2) 50 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:26:05.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:02.450308+0000 mgr.x (mgr.34775) 20 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 204 KiB/s wr, 27 op/s 2024-09-17T09:26:05.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:03.256639+0000 mon.a (mon.0) 685 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:26:05.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:03.258906+0000 mon.b (mon.2) 51 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:26:05.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:03.259120+0000 mon.b (mon.2) 52 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:26:05.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:03.258671+0000 mon.c (mon.1) 29 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:26:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:03.259594+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:26:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:04.166179+0000 mon.c (mon.1) 30 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:26:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:04.173953+0000 mon.b (mon.2) 54 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:26:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:04.450983+0000 mgr.x (mgr.34775) 21 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 241 KiB/s wr, 28 op/s 2024-09-17T09:26:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:04.672541+0000 mon.b (mon.2) 55 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:26:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:04.676384+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:26:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:04.676574+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:26:05.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:04.676784+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:26:05.547 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:03.258671+0000 mon.c (mon.1) 29 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:26:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:03.259594+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:26:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:04.166179+0000 mon.c (mon.1) 30 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:26:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:04.173953+0000 mon.b (mon.2) 54 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:26:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:04.450983+0000 mgr.x (mgr.34775) 21 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 241 KiB/s wr, 28 op/s 2024-09-17T09:26:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:04.672541+0000 mon.b (mon.2) 55 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:26:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:04.676384+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:26:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:04.676574+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:26:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:04.676784+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:26:05.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:03.258671+0000 mon.c (mon.1) 29 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:26:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:03.259594+0000 mon.b (mon.2) 53 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:26:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:04.166179+0000 mon.c (mon.1) 30 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:26:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:04.173953+0000 mon.b (mon.2) 54 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T09:26:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:04.450983+0000 mgr.x (mgr.34775) 21 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 241 KiB/s wr, 28 op/s 2024-09-17T09:26:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:04.672541+0000 mon.b (mon.2) 55 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T09:26:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:04.676384+0000 mon.b (mon.2) 56 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:26:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:04.676574+0000 mon.b (mon.2) 57 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:26:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:04.676784+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:26:06.507 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:05.352623+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:06.507 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:05.358526+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:06.508 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:05.880955+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:06.508 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:05.886845+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:06.508 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:05.352623+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:06.508 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:05.358526+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:06.508 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:05.880955+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:06.508 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:05.886845+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:06.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:05.352623+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:06.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:05.358526+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:06.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:05.880955+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:06.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:05.886845+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:07.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:06.451541+0000 mgr.x (mgr.34775) 22 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 215 KiB/s wr, 25 op/s 2024-09-17T09:26:07.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:06.451541+0000 mgr.x (mgr.34775) 22 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 215 KiB/s wr, 25 op/s 2024-09-17T09:26:07.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:06.451541+0000 mgr.x (mgr.34775) 22 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.1 KiB/s rd, 215 KiB/s wr, 25 op/s 2024-09-17T09:26:09.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:08.452056+0000 mgr.x (mgr.34775) 23 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 181 KiB/s wr, 21 op/s 2024-09-17T09:26:09.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:08.452056+0000 mgr.x (mgr.34775) 23 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 181 KiB/s wr, 21 op/s 2024-09-17T09:26:09.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:08.452056+0000 mgr.x (mgr.34775) 23 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 181 KiB/s wr, 21 op/s 2024-09-17T09:26:10.695 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:09.332234+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:10.695 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:09.339117+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:10.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:09.332234+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:10.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:09.339117+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:10.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:09.332234+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:10.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:09.339117+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:11.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:10.452776+0000 mgr.x (mgr.34775) 24 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 201 KiB/s wr, 22 op/s 2024-09-17T09:26:11.797 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:10.452776+0000 mgr.x (mgr.34775) 24 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 201 KiB/s wr, 22 op/s 2024-09-17T09:26:11.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:10.452776+0000 mgr.x (mgr.34775) 24 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.3 KiB/s rd, 201 KiB/s wr, 22 op/s 2024-09-17T09:26:13.298 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:12 smithi029 systemd[1]: Stopping Ceph node-exporter.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:26:13.298 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:13 smithi029 podman[51365]: 2024-09-17 09:26:13.126586775 +0000 UTC m=+0.123232352 container died 8202f7eb3e968b92e98e5ba915e53e7dd8ec0a47bd28a5ae7c9adaefa88523c3 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:13.798 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:13 smithi029 podman[51365]: 2024-09-17 09:26:13.614112107 +0000 UTC m=+0.610757717 container cleanup 8202f7eb3e968b92e98e5ba915e53e7dd8ec0a47bd28a5ae7c9adaefa88523c3 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:13.798 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:13 smithi029 podman[51365]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a 2024-09-17T09:26:13.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:12.453184+0000 mgr.x (mgr.34775) 25 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-17T09:26:13.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:12.590365+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:12.597682+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:12.601165+0000 mgr.x (mgr.34775) 26 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 77 KiB/s wr, 5 op/s 2024-09-17T09:26:13.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:12.604926+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:12.608953+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:12.614588+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:12.619011+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:12.626905+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:12.629289+0000 mgr.x (mgr.34775) 27 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-17T09:26:13.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:12.629523+0000 mgr.x (mgr.34775) 28 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T09:26:13.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:12.634557+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:12.720816+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:12.724552+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:12.728249+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:12.731858+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:12.732755+0000 mgr.x (mgr.34775) 29 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T09:26:13.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:12.733078+0000 mgr.x (mgr.34775) 30 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi029 2024-09-17T09:26:13.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:12.453184+0000 mgr.x (mgr.34775) 25 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-17T09:26:13.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:12.590365+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:12.597682+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:12.601165+0000 mgr.x (mgr.34775) 26 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 77 KiB/s wr, 5 op/s 2024-09-17T09:26:13.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:12.604926+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:12.608953+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:12.614588+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:12.619011+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:12.626905+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:12.629289+0000 mgr.x (mgr.34775) 27 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-17T09:26:13.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:12.629523+0000 mgr.x (mgr.34775) 28 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T09:26:13.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:12.634557+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:12.720816+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:12.724552+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:12.728249+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:12.731858+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:13.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:12.732755+0000 mgr.x (mgr.34775) 29 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T09:26:13.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:12.733078+0000 mgr.x (mgr.34775) 30 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi029 2024-09-17T09:26:14.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:12.453184+0000 mgr.x (mgr.34775) 25 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-17T09:26:14.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:12.590365+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:14.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:12.597682+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:14.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:12.601165+0000 mgr.x (mgr.34775) 26 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 77 KiB/s wr, 5 op/s 2024-09-17T09:26:14.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:12.604926+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:14.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:12.608953+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:14.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:12.614588+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:14.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:12.619011+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:14.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:12.626905+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:14.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:12.629289+0000 mgr.x (mgr.34775) 27 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-17T09:26:14.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:12.629523+0000 mgr.x (mgr.34775) 28 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T09:26:14.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:12.634557+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:14.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:12.720816+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:14.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:12.724552+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:14.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:12.728249+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:14.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:12.731858+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:14.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:12.732755+0000 mgr.x (mgr.34775) 29 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T09:26:14.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:12.733078+0000 mgr.x (mgr.34775) 30 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi029 2024-09-17T09:26:14.236 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:13 smithi029 podman[51378]: 2024-09-17 09:26:13.907262195 +0000 UTC m=+0.783833930 container remove 8202f7eb3e968b92e98e5ba915e53e7dd8ec0a47bd28a5ae7c9adaefa88523c3 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:14.236 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:13 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:26:14.236 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:13 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-17T09:26:14.237 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:13 smithi029 systemd[1]: Stopped Ceph node-exporter.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:26:14.237 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:13 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 1.317s CPU time. 2024-09-17T09:26:14.237 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:13 smithi029 systemd[1]: Starting Ceph node-exporter.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:26:14.237 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 podman[51420]: 2024-09-17 09:26:14.000815958 +0000 UTC m=+0.017225017 image pull e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 docker.io/prom/node-exporter:v0.18.1 2024-09-17T09:26:14.237 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 podman[51420]: 2024-09-17 09:26:14.11126849 +0000 UTC m=+0.127677531 container create 34bfa263169ff229618628f882c64f7d1fb220209c3d55de705b616ff310bdc5 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:14.548 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 podman[51420]: 2024-09-17 09:26:14.356975311 +0000 UTC m=+0.373384351 container init 34bfa263169ff229618628f882c64f7d1fb220209c3d55de705b616ff310bdc5 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:14.548 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 podman[51420]: 2024-09-17 09:26:14.360272352 +0000 UTC m=+0.376681394 container start 34bfa263169ff229618628f882c64f7d1fb220209c3d55de705b616ff310bdc5 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:14.548 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2024-09-17T09:26:14.548 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - arp" source="node_exporter.go:104" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - bcache" source="node_exporter.go:104" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - bonding" source="node_exporter.go:104" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - cpu" source="node_exporter.go:104" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - edac" source="node_exporter.go:104" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - entropy" source="node_exporter.go:104" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - filefd" source="node_exporter.go:104" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2024-09-17T09:26:14.549 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2024-09-17T09:26:14.550 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2024-09-17T09:26:14.550 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2024-09-17T09:26:14.550 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2024-09-17T09:26:14.550 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2024-09-17T09:26:14.550 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - netclass" source="node_exporter.go:104" 2024-09-17T09:26:14.550 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - netdev" source="node_exporter.go:104" 2024-09-17T09:26:14.550 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - netstat" source="node_exporter.go:104" 2024-09-17T09:26:14.550 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - nfs" source="node_exporter.go:104" 2024-09-17T09:26:14.550 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2024-09-17T09:26:14.551 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - pressure" source="node_exporter.go:104" 2024-09-17T09:26:14.551 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2024-09-17T09:26:14.551 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - stat" source="node_exporter.go:104" 2024-09-17T09:26:14.551 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - textfile" source="node_exporter.go:104" 2024-09-17T09:26:14.551 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - time" source="node_exporter.go:104" 2024-09-17T09:26:14.551 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - uname" source="node_exporter.go:104" 2024-09-17T09:26:14.551 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2024-09-17T09:26:14.551 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - xfs" source="node_exporter.go:104" 2024-09-17T09:26:14.551 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg=" - zfs" source="node_exporter.go:104" 2024-09-17T09:26:14.551 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a[51435]: time="2024-09-17T09:26:14Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2024-09-17T09:26:14.551 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 bash[51420]: 34bfa263169ff229618628f882c64f7d1fb220209c3d55de705b616ff310bdc5 2024-09-17T09:26:14.551 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:26:14 smithi029 systemd[1]: Started Ceph node-exporter.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:26:15.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:13.595820+0000 mon.a (mon.0) 704 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:26:15.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:14.465236+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:15.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:14.471392+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:13.595820+0000 mon.a (mon.0) 704 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:26:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:14.465236+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:14.471392+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:15.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:13.595820+0000 mon.a (mon.0) 704 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:26:15.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:14.465236+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:15.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:14.471392+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:15.907 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:14.474723+0000 mgr.x (mgr.34775) 31 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T09:26:15.907 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:14.484539+0000 mgr.x (mgr.34775) 32 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi029 2024-09-17T09:26:15.907 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:14.601944+0000 mgr.x (mgr.34775) 33 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-17T09:26:16.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:14.474723+0000 mgr.x (mgr.34775) 31 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T09:26:16.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:14.484539+0000 mgr.x (mgr.34775) 32 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi029 2024-09-17T09:26:16.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:14.601944+0000 mgr.x (mgr.34775) 33 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-17T09:26:16.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:14.474723+0000 mgr.x (mgr.34775) 31 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T09:26:16.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:14.484539+0000 mgr.x (mgr.34775) 32 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi029 2024-09-17T09:26:16.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:14.601944+0000 mgr.x (mgr.34775) 33 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-17T09:26:18.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:16.602318+0000 mgr.x (mgr.34775) 34 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-17T09:26:18.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:16.602318+0000 mgr.x (mgr.34775) 34 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-17T09:26:18.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:16.602318+0000 mgr.x (mgr.34775) 34 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-17T09:26:19.404 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:18.602953+0000 mgr.x (mgr.34775) 35 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-17T09:26:19.404 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:18.602953+0000 mgr.x (mgr.34775) 35 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-17T09:26:19.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:18.602953+0000 mgr.x (mgr.34775) 35 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-17T09:26:21.953 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:20.603531+0000 mgr.x (mgr.34775) 36 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:21.953 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:20.603531+0000 mgr.x (mgr.34775) 36 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:22.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:20.603531+0000 mgr.x (mgr.34775) 36 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:22.226 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:22 smithi029 systemd[1]: Stopping Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:26:22.548 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[42985]: level=info ts=2024-09-17T09:26:22.225Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2024-09-17T09:26:22.548 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:22 smithi029 podman[51736]: 2024-09-17 09:26:22.246010283 +0000 UTC m=+0.126861766 container died a5a9a8d22f29626ff43a834df8d8be84037bb2a0f701de2c4968bf95ddaeabd7 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:23.010 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:22 smithi029 podman[51736]: 2024-09-17 09:26:22.779279569 +0000 UTC m=+0.660131039 container cleanup a5a9a8d22f29626ff43a834df8d8be84037bb2a0f701de2c4968bf95ddaeabd7 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:23.010 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:22 smithi029 podman[51736]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a 2024-09-17T09:26:23.288 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 podman[51749]: 2024-09-17 09:26:23.009482127 +0000 UTC m=+0.760285583 container remove a5a9a8d22f29626ff43a834df8d8be84037bb2a0f701de2c4968bf95ddaeabd7 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:23.289 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 podman[51749]: 2024-09-17 09:26:23.110628442 +0000 UTC m=+0.861431899 volume remove 010fb0c79b3fb2997a1908a686a58e8c9beb8fbf278b5a0bd1cb7a3e07b801a8 2024-09-17T09:26:23.289 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T09:26:23.289 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 systemd[1]: Stopped Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:26:23.289 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 systemd[1]: Starting Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:26:23.548 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 podman[51791]: 2024-09-17 09:26:23.287732184 +0000 UTC m=+0.101221319 volume create 40a502cae9ec4d68032d8de3f6a8d02cecd9e64afd5ecfec83ce3102a2814173 2024-09-17T09:26:23.548 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 podman[51791]: 2024-09-17 09:26:23.204089799 +0000 UTC m=+0.017578942 image pull 0881eb8f169f5556a292b4e2c01d683172b12830a62a9225a98a8e206bb734f0 docker.io/prom/alertmanager:v0.20.0 2024-09-17T09:26:23.548 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 podman[51791]: 2024-09-17 09:26:23.380117408 +0000 UTC m=+0.193606542 container create 7bd158a6d65c037fc5b6057c7ae4511ae9d0a62ba41f3da07241c94162a0e93e (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:24.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:22.604046+0000 mgr.x (mgr.34775) 37 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:24.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:22.604046+0000 mgr.x (mgr.34775) 37 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:24.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:22.604046+0000 mgr.x (mgr.34775) 37 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:24.048 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 podman[51791]: 2024-09-17 09:26:23.746269361 +0000 UTC m=+0.559758503 container init 7bd158a6d65c037fc5b6057c7ae4511ae9d0a62ba41f3da07241c94162a0e93e (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:24.048 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 podman[51791]: 2024-09-17 09:26:23.749722099 +0000 UTC m=+0.563211236 container start 7bd158a6d65c037fc5b6057c7ae4511ae9d0a62ba41f3da07241c94162a0e93e (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:24.048 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[51806]: level=info ts=2024-09-17T09:26:23.772Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2024-09-17T09:26:24.049 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[51806]: level=info ts=2024-09-17T09:26:23.772Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2024-09-17T09:26:24.049 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[51806]: level=info ts=2024-09-17T09:26:23.773Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.29 port=9094 2024-09-17T09:26:24.049 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[51806]: level=info ts=2024-09-17T09:26:23.774Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T09:26:24.049 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[51806]: level=info ts=2024-09-17T09:26:23.794Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T09:26:24.049 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[51806]: level=info ts=2024-09-17T09:26:23.794Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T09:26:24.049 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[51806]: level=info ts=2024-09-17T09:26:23.797Z caller=main.go:497 msg=Listening address=:9093 2024-09-17T09:26:24.049 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 bash[51791]: 7bd158a6d65c037fc5b6057c7ae4511ae9d0a62ba41f3da07241c94162a0e93e 2024-09-17T09:26:24.049 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:23 smithi029 systemd[1]: Started Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:26:25.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:23.885162+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:25.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:23.893177+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:25.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:23.895472+0000 mon.b (mon.2) 59 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:26:25.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:23.896050+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:26:25.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:23.900530+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:26:25.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:23.885162+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:25.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:23.893177+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:25.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:23.895472+0000 mon.b (mon.2) 59 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:26:25.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:23.896050+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:26:25.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:23.900530+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:26:25.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:23.885162+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:25.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:23.893177+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:25.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:23.895472+0000 mon.b (mon.2) 59 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:26:25.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:23.896050+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:26:25.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:23.900530+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:26:26.048 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[51806]: level=info ts=2024-09-17T09:26:25.774Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000074619s 2024-09-17T09:26:26.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:23.895099+0000 mgr.x (mgr.34775) 38 : cephadm [INF] Reconfiguring iscsi.foo.smithi029.gellkh (dependencies changed)... 2024-09-17T09:26:26.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:23.901401+0000 mgr.x (mgr.34775) 39 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:26:26.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:24.604872+0000 mgr.x (mgr.34775) 40 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:26.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:23.895099+0000 mgr.x (mgr.34775) 38 : cephadm [INF] Reconfiguring iscsi.foo.smithi029.gellkh (dependencies changed)... 2024-09-17T09:26:26.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:23.901401+0000 mgr.x (mgr.34775) 39 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:26:26.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:24.604872+0000 mgr.x (mgr.34775) 40 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:26.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:23.895099+0000 mgr.x (mgr.34775) 38 : cephadm [INF] Reconfiguring iscsi.foo.smithi029.gellkh (dependencies changed)... 2024-09-17T09:26:26.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:23.901401+0000 mgr.x (mgr.34775) 39 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:26:26.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:24.604872+0000 mgr.x (mgr.34775) 40 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:27.121 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:26.605354+0000 mgr.x (mgr.34775) 41 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:27.121 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:26.605354+0000 mgr.x (mgr.34775) 41 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:27.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:26.605354+0000 mgr.x (mgr.34775) 41 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:28.392 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:27.263990+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:28.393 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:27.269688+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:28.393 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:27.271449+0000 mgr.x (mgr.34775) 42 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:26:28.393 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:27.495921+0000 mgr.x (mgr.34775) 43 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:26:28.393 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:27.720587+0000 mon.a (mon.0) 712 : audit [DBG] from='client.? 172.21.15.29:0/436192128' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:26:28.393 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:28.055226+0000 mon.c (mon.1) 31 : audit [INF] from='client.? 172.21.15.29:0/1324708493' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: access denied 2024-09-17T09:26:28.393 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:27.263990+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:28.393 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:27.269688+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:28.393 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:27.271449+0000 mgr.x (mgr.34775) 42 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:26:28.393 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:27.495921+0000 mgr.x (mgr.34775) 43 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:26:28.394 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:27.720587+0000 mon.a (mon.0) 712 : audit [DBG] from='client.? 172.21.15.29:0/436192128' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:26:28.394 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:28 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:28.055226+0000 mon.c (mon.1) 31 : audit [INF] from='client.? 172.21.15.29:0/1324708493' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: access denied 2024-09-17T09:26:28.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:27.263990+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:28.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:27.269688+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:28.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:27.271449+0000 mgr.x (mgr.34775) 42 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:26:28.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:27.495921+0000 mgr.x (mgr.34775) 43 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:26:28.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:27.720587+0000 mon.a (mon.0) 712 : audit [DBG] from='client.? 172.21.15.29:0/436192128' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:26:28.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:28.055226+0000 mon.c (mon.1) 31 : audit [INF] from='client.? 172.21.15.29:0/1324708493' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: access denied 2024-09-17T09:26:29.127 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:26:29.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:28.605908+0000 mgr.x (mgr.34775) 44 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:28.605908+0000 mgr.x (mgr.34775) 44 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:29.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:28.605908+0000 mgr.x (mgr.34775) 44 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:29.785 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:26:29.785 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 starting - - - - 2024-09-17T09:26:29.785 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (4m) 30s ago 4m 35.3M - 6.7.4 557c83e11646 b74e78c8a6b5 2024-09-17T09:26:29.785 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 starting - - - - 2024-09-17T09:26:29.785 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443 running (43s) 30s ago 8m 459M - 18.2.4-855-g19e2a2be 980a0bfcd684 d35594bdef28 2024-09-17T09:26:29.785 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:9283 running (10m) 24s ago 9m 113M - 16.2.0 24ecd6d5f14c 9d130c67b36d 2024-09-17T09:26:29.785 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (10m) 24s ago 10m 50.2M 2048M 16.2.0 24ecd6d5f14c f3476652dfdd 2024-09-17T09:26:29.785 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (8m) 30s ago 8m 39.5M 2048M 16.2.0 24ecd6d5f14c 203d1f26d7b8 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (9m) 24s ago 9m 50.1M 2048M 16.2.0 24ecd6d5f14c aa05f3e5656a 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 starting - - - - 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (5m) 30s ago 5m 16.9M - 0.18.1 e5a616e4b9cf 0e6931583242 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (8m) 24s ago 8m 44.8M 4096M 16.2.0 24ecd6d5f14c 991731ce08b5 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (8m) 24s ago 8m 44.9M 4096M 16.2.0 24ecd6d5f14c d5bce63fadc8 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (7m) 24s ago 7m 41.5M 4096M 16.2.0 24ecd6d5f14c e9678d2f1685 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (7m) 24s ago 7m 48.3M 4096M 16.2.0 24ecd6d5f14c dcf7122a7d99 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (7m) 30s ago 7m 47.7M 4096M 16.2.0 24ecd6d5f14c 048c13025d32 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (6m) 30s ago 6m 46.3M 4096M 16.2.0 24ecd6d5f14c 10a64919fbc9 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (6m) 30s ago 6m 43.0M 4096M 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (6m) 30s ago 6m 48.7M 4096M 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 running (4m) 30s ago 5m 27.0M - 2.18.1 de242295e225 94209ce0b68f 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (4m) 24s ago 4m 38.9M - 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (4m) 30s ago 4m 38.8M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:26:29.786 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (3m) 24s ago 3m 36.4M - 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:26:29.787 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (3m) 30s ago 3m 36.0M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:26:30.183 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:26:30.183 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:26:30.183 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2024-09-17T09:26:30.183 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:26:30.183 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:26:30.184 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2024-09-17T09:26:30.184 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T09:26:30.184 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:26:30.184 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:26:30.184 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T09:26:30.184 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:26:30.184 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": {}, 2024-09-17T09:26:30.184 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:26:30.184 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:26:30.184 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:26:30.184 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:26:30.184 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 16, 2024-09-17T09:26:30.185 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T09:26:30.185 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:26:30.185 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:26:30.521 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:26:30.521 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:26:30.521 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:26:30.521 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:26:30.521 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [], 2024-09-17T09:26:30.521 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "1/23 daemons upgraded", 2024-09-17T09:26:30.521 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "", 2024-09-17T09:26:30.521 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:26:30.521 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:26:30.522 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:29.112711+0000 mgr.x (mgr.34775) 45 : audit [DBG] from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:26:30.522 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:29.451569+0000 mgr.x (mgr.34775) 46 : audit [DBG] from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:26:30.522 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:29.776664+0000 mgr.x (mgr.34775) 47 : audit [DBG] from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:26:30.522 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:30.182803+0000 mon.c (mon.1) 32 : audit [DBG] from='client.? 172.21.15.29:0/1155551303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:26:30.522 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:29.112711+0000 mgr.x (mgr.34775) 45 : audit [DBG] from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:26:30.522 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:29.451569+0000 mgr.x (mgr.34775) 46 : audit [DBG] from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:26:30.522 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:29.776664+0000 mgr.x (mgr.34775) 47 : audit [DBG] from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:26:30.522 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:30 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:30.182803+0000 mon.c (mon.1) 32 : audit [DBG] from='client.? 172.21.15.29:0/1155551303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:26:30.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:29.112711+0000 mgr.x (mgr.34775) 45 : audit [DBG] from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:26:30.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:29.451569+0000 mgr.x (mgr.34775) 46 : audit [DBG] from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:26:30.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:29.776664+0000 mgr.x (mgr.34775) 47 : audit [DBG] from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:26:30.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:30.182803+0000 mon.c (mon.1) 32 : audit [DBG] from='client.? 172.21.15.29:0/1155551303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:26:30.913 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T09:26:30.913 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T09:26:30.913 INFO:teuthology.orchestra.run.smithi029.stdout: daemon iscsi.foo.smithi029.gellkh on smithi029 is in error state 2024-09-17T09:26:31.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:30.519689+0000 mgr.x (mgr.34775) 48 : audit [DBG] from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:26:31.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:30.606197+0000 mgr.x (mgr.34775) 49 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:31.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:30.912918+0000 mon.c (mon.1) 33 : audit [DBG] from='client.? 172.21.15.29:0/3377025213' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:26:31.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:30.519689+0000 mgr.x (mgr.34775) 48 : audit [DBG] from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:26:31.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:30.606197+0000 mgr.x (mgr.34775) 49 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:31.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:30.912918+0000 mon.c (mon.1) 33 : audit [DBG] from='client.? 172.21.15.29:0/3377025213' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:26:31.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:30.519689+0000 mgr.x (mgr.34775) 48 : audit [DBG] from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:26:31.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:30.606197+0000 mgr.x (mgr.34775) 49 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:31.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:30.912918+0000 mon.c (mon.1) 33 : audit [DBG] from='client.? 172.21.15.29:0/3377025213' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:26:33.953 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:32.606662+0000 mgr.x (mgr.34775) 50 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:32.606662+0000 mgr.x (mgr.34775) 50 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:34.048 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:26:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[51806]: level=info ts=2024-09-17T09:26:33.774Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000613459s 2024-09-17T09:26:34.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:32.606662+0000 mgr.x (mgr.34775) 50 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:36.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:34.607598+0000 mgr.x (mgr.34775) 51 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:36.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:34.607598+0000 mgr.x (mgr.34775) 51 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:36.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:34.607598+0000 mgr.x (mgr.34775) 51 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:36.658 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 systemd[1]: Stopping Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:26:36.659 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=warn ts=2024-09-17T09:26:36.655Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2024-09-17T09:26:36.659 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:26:36.656Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:26:36.659 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:26:36.656Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:26:36.659 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:26:36.656Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:26:36.659 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:26:36.656Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:26:36.659 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:26:36.656Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:26:36.659 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:26:36.656Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:26:36.659 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:26:36.656Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:26:36.659 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:26:36.657Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:26:36.659 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:26:36.657Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:26:36.659 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:26:36.658Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:26:36.659 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[37166]: level=info ts=2024-09-17T09:26:36.658Z caller=main.go:761 msg="See you next time!" 2024-09-17T09:26:36.966 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:36 smithi103 podman[43434]: 2024-09-17 09:26:36.666054931 +0000 UTC m=+0.125183590 container died 94209ce0b68f7a5d633f4b2d0959e023b00803b892b553d735137e9a44ed7142 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:37.255 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 podman[43434]: 2024-09-17 09:26:37.055042302 +0000 UTC m=+0.514170950 container cleanup 94209ce0b68f7a5d633f4b2d0959e023b00803b892b553d735137e9a44ed7142 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:37.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 podman[43434]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a 2024-09-17T09:26:37.597 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 podman[43446]: 2024-09-17 09:26:37.283645822 +0000 UTC m=+0.617422808 container remove 94209ce0b68f7a5d633f4b2d0959e023b00803b892b553d735137e9a44ed7142 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:37.597 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T09:26:37.597 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:26:37.597 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:26:37.597 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 podman[43491]: 2024-09-17 09:26:37.371590669 +0000 UTC m=+0.017510070 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:26:37.597 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 podman[43491]: 2024-09-17 09:26:37.482204365 +0000 UTC m=+0.128123751 container create 4f595e3514cea62e3c473dc4f05afd959e1af63a086431cdbb540b4a403827a3 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:38.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:36.607956+0000 mgr.x (mgr.34775) 52 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:38.011 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 podman[43491]: 2024-09-17 09:26:37.711662223 +0000 UTC m=+0.357581610 container init 4f595e3514cea62e3c473dc4f05afd959e1af63a086431cdbb540b4a403827a3 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:38.012 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 podman[43491]: 2024-09-17 09:26:37.714588673 +0000 UTC m=+0.360508061 container start 4f595e3514cea62e3c473dc4f05afd959e1af63a086431cdbb540b4a403827a3 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:38.012 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.744Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:26:38.012 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.744Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:26:38.012 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.744Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:26:38.012 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.744Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:26:38.012 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.744Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:26:38.012 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.744Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:26:38.012 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.746Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:26:38.012 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.746Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:26:38.012 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.748Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:26:38.012 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.749Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-17T09:26:38.012 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.755Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-17T09:26:38.013 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.755Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-17T09:26:38.013 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.755Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=6.830882ms 2024-09-17T09:26:38.013 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.757Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:26:38.013 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.757Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:26:38.013 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.757Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:26:38.013 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.757Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:26:38.013 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.757Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:26:38.013 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.757Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:26:38.013 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.757Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:26:38.013 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.757Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:26:38.013 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.757Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:26:38.013 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.757Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:26:38.013 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.758Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:26:38.014 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.758Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:26:38.014 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=info ts=2024-09-17T09:26:37.758Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:26:38.014 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43506]: level=error ts=2024-09-17T09:26:37.758Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:26:38.014 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 bash[43491]: 4f595e3514cea62e3c473dc4f05afd959e1af63a086431cdbb540b4a403827a3 2024-09-17T09:26:38.014 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:26:38.014 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:37 smithi103 podman[43523]: 2024-09-17 09:26:37.793678569 +0000 UTC m=+0.024667828 container died 4f595e3514cea62e3c473dc4f05afd959e1af63a086431cdbb540b4a403827a3 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:38.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:36.607956+0000 mgr.x (mgr.34775) 52 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:38.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:36.607956+0000 mgr.x (mgr.34775) 52 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:38.367 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:38 smithi103 systemd[1]: Stopping Ceph node-exporter.b for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:26:38.367 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:38 smithi103 podman[43523]: 2024-09-17 09:26:38.366476806 +0000 UTC m=+0.597466065 container remove 4f595e3514cea62e3c473dc4f05afd959e1af63a086431cdbb540b4a403827a3 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:38.755 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:38 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:26:38.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:38 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:26:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:37.812482+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:37.818136+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:37.819270+0000 mgr.x (mgr.34775) 53 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T09:26:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:37.819687+0000 mgr.x (mgr.34775) 54 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi103 2024-09-17T09:26:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:37.812482+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:37.818136+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:37.819270+0000 mgr.x (mgr.34775) 53 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T09:26:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:37.819687+0000 mgr.x (mgr.34775) 54 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi103 2024-09-17T09:26:39.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:37.812482+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:39.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:37.818136+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:39.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:37.819270+0000 mgr.x (mgr.34775) 53 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T09:26:39.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:37.819687+0000 mgr.x (mgr.34775) 54 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi103 2024-09-17T09:26:39.755 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:39 smithi103 podman[43632]: 2024-09-17 09:26:39.481619277 +0000 UTC m=+1.326033091 container died 0e69315832421c42e8ec61fc749374afdd7e280afafe7734c1105e40405ef391 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T09:26:40.119 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:38.608831+0000 mgr.x (mgr.34775) 55 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:40.119 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:38.608831+0000 mgr.x (mgr.34775) 55 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:40.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:38.608831+0000 mgr.x (mgr.34775) 55 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:40.755 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:40 smithi103 podman[43632]: 2024-09-17 09:26:40.555446277 +0000 UTC m=+2.399860084 container cleanup 0e69315832421c42e8ec61fc749374afdd7e280afafe7734c1105e40405ef391 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T09:26:40.756 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:40 smithi103 podman[43632]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b 2024-09-17T09:26:41.130 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:40 smithi103 podman[43648]: 2024-09-17 09:26:40.840431471 +0000 UTC m=+1.362026539 container remove 0e69315832421c42e8ec61fc749374afdd7e280afafe7734c1105e40405ef391 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T09:26:41.130 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:40 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:26:41.130 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:40 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-17T09:26:41.130 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:40 smithi103 systemd[1]: Stopped Ceph node-exporter.b for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:26:41.130 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:40 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 1.242s CPU time. 2024-09-17T09:26:41.130 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:40 smithi103 systemd[1]: Starting Ceph node-exporter.b for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:26:41.131 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 podman[43702]: 2024-09-17 09:26:40.924357199 +0000 UTC m=+0.016035417 image pull e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 docker.io/prom/node-exporter:v0.18.1 2024-09-17T09:26:41.131 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 podman[43702]: 2024-09-17 09:26:41.033509812 +0000 UTC m=+0.125188086 container create 1adf1b5baad1b8aba6d426b76683866f49a51dbdb8219b0f7d6e10a61e6ffd4d (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T09:26:41.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 podman[43702]: 2024-09-17 09:26:41.236389726 +0000 UTC m=+0.328067949 container init 1adf1b5baad1b8aba6d426b76683866f49a51dbdb8219b0f7d6e10a61e6ffd4d (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T09:26:41.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 podman[43702]: 2024-09-17 09:26:41.239709223 +0000 UTC m=+0.331387441 container start 1adf1b5baad1b8aba6d426b76683866f49a51dbdb8219b0f7d6e10a61e6ffd4d (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T09:26:41.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2024-09-17T09:26:41.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2024-09-17T09:26:41.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2024-09-17T09:26:41.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - arp" source="node_exporter.go:104" 2024-09-17T09:26:41.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - bcache" source="node_exporter.go:104" 2024-09-17T09:26:41.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - bonding" source="node_exporter.go:104" 2024-09-17T09:26:41.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2024-09-17T09:26:41.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - cpu" source="node_exporter.go:104" 2024-09-17T09:26:41.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2024-09-17T09:26:41.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2024-09-17T09:26:41.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - edac" source="node_exporter.go:104" 2024-09-17T09:26:41.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - entropy" source="node_exporter.go:104" 2024-09-17T09:26:41.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - filefd" source="node_exporter.go:104" 2024-09-17T09:26:41.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2024-09-17T09:26:41.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2024-09-17T09:26:41.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2024-09-17T09:26:41.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2024-09-17T09:26:41.507 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2024-09-17T09:26:41.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2024-09-17T09:26:41.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2024-09-17T09:26:41.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - netclass" source="node_exporter.go:104" 2024-09-17T09:26:41.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - netdev" source="node_exporter.go:104" 2024-09-17T09:26:41.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - netstat" source="node_exporter.go:104" 2024-09-17T09:26:41.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - nfs" source="node_exporter.go:104" 2024-09-17T09:26:41.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2024-09-17T09:26:41.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - pressure" source="node_exporter.go:104" 2024-09-17T09:26:41.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2024-09-17T09:26:41.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - stat" source="node_exporter.go:104" 2024-09-17T09:26:41.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - textfile" source="node_exporter.go:104" 2024-09-17T09:26:41.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - time" source="node_exporter.go:104" 2024-09-17T09:26:41.508 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - uname" source="node_exporter.go:104" 2024-09-17T09:26:41.509 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2024-09-17T09:26:41.509 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - xfs" source="node_exporter.go:104" 2024-09-17T09:26:41.509 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg=" - zfs" source="node_exporter.go:104" 2024-09-17T09:26:41.509 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b[43716]: time="2024-09-17T09:26:41Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2024-09-17T09:26:41.509 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 bash[43702]: 1adf1b5baad1b8aba6d426b76683866f49a51dbdb8219b0f7d6e10a61e6ffd4d 2024-09-17T09:26:41.509 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:26:41 smithi103 systemd[1]: Started Ceph node-exporter.b for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:26:41.993 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:40.609266+0000 mgr.x (mgr.34775) 56 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:41.993 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:41.212543+0000 mon.c (mon.1) 34 : audit [DBG] from='client.? 172.21.15.29:0/2686698660' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:26:41.993 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:41.337306+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:41.993 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:41.343735+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:41.993 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:41.373774+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:41.993 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:41.377600+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:41.993 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:41.379662+0000 mon.b (mon.2) 61 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:26:41.993 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:41.553817+0000 mon.a (mon.0) 719 : audit [INF] from='client.? 172.21.15.29:0/644470213' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: access denied 2024-09-17T09:26:41.994 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:40.609266+0000 mgr.x (mgr.34775) 56 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:41.994 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:41.212543+0000 mon.c (mon.1) 34 : audit [DBG] from='client.? 172.21.15.29:0/2686698660' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:26:41.994 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:41.337306+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:41.994 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:41.343735+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:41.994 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:41.373774+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:41.994 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:41.377600+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:41.994 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:41.379662+0000 mon.b (mon.2) 61 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:26:41.994 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:41.553817+0000 mon.a (mon.0) 719 : audit [INF] from='client.? 172.21.15.29:0/644470213' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: access denied 2024-09-17T09:26:42.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:40.609266+0000 mgr.x (mgr.34775) 56 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:42.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:41.212543+0000 mon.c (mon.1) 34 : audit [DBG] from='client.? 172.21.15.29:0/2686698660' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:26:42.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:41.337306+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:42.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:41.343735+0000 mon.a (mon.0) 716 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:42.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:41.373774+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:42.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:41.377600+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:42.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:41.379662+0000 mon.b (mon.2) 61 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:26:42.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:41.553817+0000 mon.a (mon.0) 719 : audit [INF] from='client.? 172.21.15.29:0/644470213' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: access denied 2024-09-17T09:26:43.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:42 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:41.345277+0000 mgr.x (mgr.34775) 57 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-09-17T09:26:43.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:42 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:41.353281+0000 mgr.x (mgr.34775) 58 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T09:26:43.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:42 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:41.380006+0000 mgr.x (mgr.34775) 59 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:26:43.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:42 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:41.381661+0000 mgr.x (mgr.34775) 60 : cephadm [INF] Reconfiguring daemon grafana.a on smithi103 2024-09-17T09:26:43.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:41.345277+0000 mgr.x (mgr.34775) 57 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-09-17T09:26:43.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:41.353281+0000 mgr.x (mgr.34775) 58 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T09:26:43.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:41.380006+0000 mgr.x (mgr.34775) 59 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:26:43.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:41.381661+0000 mgr.x (mgr.34775) 60 : cephadm [INF] Reconfiguring daemon grafana.a on smithi103 2024-09-17T09:26:43.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:41.345277+0000 mgr.x (mgr.34775) 57 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2024-09-17T09:26:43.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:41.353281+0000 mgr.x (mgr.34775) 58 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T09:26:43.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:41.380006+0000 mgr.x (mgr.34775) 59 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T09:26:43.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:41.381661+0000 mgr.x (mgr.34775) 60 : cephadm [INF] Reconfiguring daemon grafana.a on smithi103 2024-09-17T09:26:44.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:42.609613+0000 mgr.x (mgr.34775) 61 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:44.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:42.609613+0000 mgr.x (mgr.34775) 61 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:44.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:42.609613+0000 mgr.x (mgr.34775) 61 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:46.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:44.610296+0000 mgr.x (mgr.34775) 62 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:46.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:44.610296+0000 mgr.x (mgr.34775) 62 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:46.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:44.610296+0000 mgr.x (mgr.34775) 62 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:48.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:46.610713+0000 mgr.x (mgr.34775) 63 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:48.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:46.610713+0000 mgr.x (mgr.34775) 63 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:48.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:46.610713+0000 mgr.x (mgr.34775) 63 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:48.906 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:48 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 1. 2024-09-17T09:26:48.906 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:48 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:26:48.906 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:48 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:26:49.255 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 podman[43868]: 2024-09-17 09:26:49.019980514 +0000 UTC m=+0.197251867 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:26:49.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 podman[43868]: 2024-09-17 09:26:49.146712114 +0000 UTC m=+0.323983466 container create d4fd2db1a53202316527c4620eecebc2505ee9a9aae15c14ea03af121c6b36d9 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:49.516 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:48.611507+0000 mgr.x (mgr.34775) 64 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:49.516 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 podman[43868]: 2024-09-17 09:26:49.482920862 +0000 UTC m=+0.660192222 container init d4fd2db1a53202316527c4620eecebc2505ee9a9aae15c14ea03af121c6b36d9 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:49.517 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 podman[43868]: 2024-09-17 09:26:49.485826619 +0000 UTC m=+0.663097963 container start d4fd2db1a53202316527c4620eecebc2505ee9a9aae15c14ea03af121c6b36d9 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:49.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:48.611507+0000 mgr.x (mgr.34775) 64 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:49.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:48.611507+0000 mgr.x (mgr.34775) 64 : cluster [DBG] pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:49.932 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.515Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:26:49.932 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.515Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:26:49.932 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.515Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:26:49.932 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.515Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:26:49.933 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.515Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:26:49.933 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.515Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:26:49.933 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.517Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:26:49.933 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.517Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:26:49.933 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.520Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:26:49.933 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.520Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-17T09:26:49.934 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.526Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-17T09:26:49.934 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.526Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-17T09:26:49.934 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.526Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-17T09:26:49.934 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.526Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=6.416882ms 2024-09-17T09:26:49.934 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.528Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:26:49.934 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.528Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:26:49.934 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.528Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:26:49.935 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.528Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:26:49.935 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.528Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:26:49.935 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.528Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:26:49.935 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.528Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:26:49.935 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.528Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:26:49.935 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.528Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:26:49.936 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.528Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:26:49.936 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.528Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:26:49.936 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.528Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:26:49.936 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=info ts=2024-09-17T09:26:49.528Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:26:49.936 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[43888]: level=error ts=2024-09-17T09:26:49.529Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:26:49.936 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 bash[43868]: d4fd2db1a53202316527c4620eecebc2505ee9a9aae15c14ea03af121c6b36d9 2024-09-17T09:26:49.937 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:26:49.937 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:49 smithi103 podman[43917]: 2024-09-17 09:26:49.585651733 +0000 UTC m=+0.049556506 container died d4fd2db1a53202316527c4620eecebc2505ee9a9aae15c14ea03af121c6b36d9 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:50.255 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:50 smithi103 podman[43917]: 2024-09-17 09:26:50.128345338 +0000 UTC m=+0.592250111 container remove d4fd2db1a53202316527c4620eecebc2505ee9a9aae15c14ea03af121c6b36d9 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:26:50.255 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:50 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:26:50.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:26:50 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:26:51.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:50.611841+0000 mgr.x (mgr.34775) 65 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:51.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:51 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:50.611841+0000 mgr.x (mgr.34775) 65 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:52.004 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:50.611841+0000 mgr.x (mgr.34775) 65 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:52.256 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 systemd[1]: Stopping Ceph grafana.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:26:52.256 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:26:52+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-09-17T09:26:52.256 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:26:52+0000 lvl=info msg="Stopped RemoteCache" logger=server reason="context canceled" 2024-09-17T09:26:52.256 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:26:52+0000 lvl=info msg="Stopped provisioningServiceImpl" logger=server reason="context canceled" 2024-09-17T09:26:52.256 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:26:52+0000 lvl=info msg="Stopped BackendPluginManager" logger=server reason="context canceled" 2024-09-17T09:26:52.256 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:26:52+0000 lvl=info msg="Stopped Stream Manager" 2024-09-17T09:26:52.256 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:26:52+0000 lvl=info msg="Stopped CleanUpService" logger=server reason="context canceled" 2024-09-17T09:26:52.256 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:26:52+0000 lvl=info msg="Stopped AlertEngine" logger=server reason="context canceled" 2024-09-17T09:26:52.257 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:26:52+0000 lvl=info msg="Stopped UsageStatsService" logger=server reason="context canceled" 2024-09-17T09:26:52.257 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:26:52+0000 lvl=info msg="Stopped PluginManager" logger=server reason="context canceled" 2024-09-17T09:26:52.257 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:26:52+0000 lvl=info msg="Stopped UserAuthTokenService" logger=server reason="context canceled" 2024-09-17T09:26:52.257 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:26:52+0000 lvl=info msg="Stopped NotificationService" logger=server reason="context canceled" 2024-09-17T09:26:52.257 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[36523]: t=2024-09-17T09:26:52+0000 lvl=info msg="Stopped InternalMetricsService" logger=server reason="context canceled" 2024-09-17T09:26:52.257 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 podman[43987]: 2024-09-17 09:26:52.212811558 +0000 UTC m=+0.133509521 container died b74e78c8a6b5eee4540b90dafd8f6a324f70299107877e8d6c16637307145930 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.schema-version=1.0, io.buildah.version=1.19.8, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, maintainer=Paul Cuzner , org.label-schema.license=GPLv2, description=Ceph Grafana Container) 2024-09-17T09:26:52.755 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 podman[43987]: 2024-09-17 09:26:52.57719371 +0000 UTC m=+0.497891659 container cleanup b74e78c8a6b5eee4540b90dafd8f6a324f70299107877e8d6c16637307145930 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Paul Cuzner , org.label-schema.license=GPLv2, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, io.buildah.version=1.19.8, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image) 2024-09-17T09:26:52.756 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 podman[43987]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a 2024-09-17T09:26:53.166 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 podman[43999]: 2024-09-17 09:26:52.845520507 +0000 UTC m=+0.635264026 container remove b74e78c8a6b5eee4540b90dafd8f6a324f70299107877e8d6c16637307145930 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a, org.label-schema.schema-version=1.0, io.buildah.version=1.19.8, description=Ceph Grafana Container, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , org.label-schema.build-date=20201204) 2024-09-17T09:26:53.166 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-17T09:26:53.167 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 systemd[1]: Stopped Ceph grafana.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:26:53.167 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:52 smithi103 systemd[1]: Starting Ceph grafana.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:26:53.167 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 podman[44043]: 2024-09-17 09:26:52.933594548 +0000 UTC m=+0.016124205 image pull 557c83e11646f123a27b5e4b62ac6c45e7bb8b2e90d6044034d0db5b7019415c docker.io/ceph/ceph-grafana:6.7.4 2024-09-17T09:26:53.167 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 podman[44043]: 2024-09-17 09:26:53.05521676 +0000 UTC m=+0.137746446 container create e361816f7c1489fb4a6d6bd4714b3bacde0cca491f2057463ea954ac85f4459e (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a, io.buildah.version=1.19.8, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Paul Cuzner ) 2024-09-17T09:26:53.506 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 podman[44043]: 2024-09-17 09:26:53.290368939 +0000 UTC m=+0.372898585 container init e361816f7c1489fb4a6d6bd4714b3bacde0cca491f2057463ea954ac85f4459e (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a, org.label-schema.build-date=20201204, maintainer=Paul Cuzner , org.label-schema.license=GPLv2, description=Ceph Grafana Container, io.buildah.version=1.19.8, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0) 2024-09-17T09:26:53.506 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 podman[44043]: 2024-09-17 09:26:53.293218907 +0000 UTC m=+0.375748558 container start e361816f7c1489fb4a6d6bd4714b3bacde0cca491f2057463ea954ac85f4459e (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a, maintainer=Paul Cuzner , org.label-schema.schema-version=1.0, description=Ceph Grafana Container, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.19.8, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-09-17T09:26:53.506 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000 2024-09-17T09:26:53.506 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-09-17T09:26:53.506 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-09-17T09:26:53.506 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-17T09:26:53.506 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-17T09:26:53.506 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-17T09:26:53.507 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-17T09:26:53.507 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-09-17T09:26:53.507 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-09-17T09:26:53.507 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-09-17T09:26:53.507 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-09-17T09:26:53.507 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-09-17T09:26:53.507 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="App mode production" logger=settings 2024-09-17T09:26:53.507 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing SqlStore" logger=server 2024-09-17T09:26:53.508 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-09-17T09:26:53.508 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Starting DB migration" logger=migrator 2024-09-17T09:26:53.509 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-09-17T09:26:53.509 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-09-17T09:26:53.509 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-09-17T09:26:53.509 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-09-17T09:26:53.509 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-09-17T09:26:53.509 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-09-17T09:26:53.509 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-09-17T09:26:53.509 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-09-17T09:26:53.509 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-09-17T09:26:53.509 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-09-17T09:26:53.509 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-09-17T09:26:53.510 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-09-17T09:26:53.510 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-09-17T09:26:53.510 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-09-17T09:26:53.510 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-09-17T09:26:53.510 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-09-17T09:26:53.510 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-09-17T09:26:53.510 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-09-17T09:26:53.510 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-09-17T09:26:53.510 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-09-17T09:26:53.510 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-09-17T09:26:53.510 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-09-17T09:26:53.510 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-09-17T09:26:53.511 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-09-17T09:26:53.511 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-09-17T09:26:53.511 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-09-17T09:26:53.511 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-09-17T09:26:53.511 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-09-17T09:26:53.511 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-09-17T09:26:53.511 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-09-17T09:26:53.511 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-09-17T09:26:53.511 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-09-17T09:26:53.511 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-09-17T09:26:53.511 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-09-17T09:26:53.511 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-09-17T09:26:53.512 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-09-17T09:26:53.512 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-09-17T09:26:53.512 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-09-17T09:26:53.512 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-17T09:26:53.512 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-09-17T09:26:53.512 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-09-17T09:26:53.512 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-09-17T09:26:53.512 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-09-17T09:26:53.512 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-09-17T09:26:53.512 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-09-17T09:26:53.512 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-09-17T09:26:53.512 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-09-17T09:26:53.513 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-09-17T09:26:53.513 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-09-17T09:26:53.513 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-09-17T09:26:53.513 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-09-17T09:26:53.513 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-09-17T09:26:53.513 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-09-17T09:26:53.513 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-09-17T09:26:53.513 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-09-17T09:26:53.513 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-09-17T09:26:53.513 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-09-17T09:26:53.513 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-09-17T09:26:53.513 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-09-17T09:26:53.514 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-09-17T09:26:53.514 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-09-17T09:26:53.514 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-09-17T09:26:53.514 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-09-17T09:26:53.514 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-17T09:26:53.514 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-09-17T09:26:53.514 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-17T09:26:53.514 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-09-17T09:26:53.514 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-17T09:26:53.514 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-17T09:26:53.514 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-09-17T09:26:53.514 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-09-17T09:26:53.514 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-09-17T09:26:53.515 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-09-17T09:26:53.515 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-09-17T09:26:53.515 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-09-17T09:26:53.515 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-09-17T09:26:53.515 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-09-17T09:26:53.515 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-09-17T09:26:53.515 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-09-17T09:26:53.515 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-09-17T09:26:53.515 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-09-17T09:26:53.515 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-09-17T09:26:53.515 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-09-17T09:26:53.515 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-09-17T09:26:53.515 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-09-17T09:26:53.516 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-09-17T09:26:53.516 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-09-17T09:26:53.516 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-09-17T09:26:53.516 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-09-17T09:26:53.516 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-09-17T09:26:53.516 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-09-17T09:26:53.516 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-09-17T09:26:53.516 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-09-17T09:26:53.516 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-09-17T09:26:53.516 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-09-17T09:26:53.516 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-09-17T09:26:53.516 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-09-17T09:26:53.517 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-09-17T09:26:53.517 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-09-17T09:26:53.517 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-09-17T09:26:53.517 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-09-17T09:26:53.517 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-09-17T09:26:53.517 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-09-17T09:26:53.517 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-09-17T09:26:53.517 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-09-17T09:26:53.517 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-09-17T09:26:53.517 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-09-17T09:26:53.517 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-09-17T09:26:53.517 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-09-17T09:26:53.518 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-09-17T09:26:53.518 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-17T09:26:53.518 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-17T09:26:53.518 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-09-17T09:26:53.518 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-09-17T09:26:53.518 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-09-17T09:26:53.518 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-09-17T09:26:53.518 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-17T09:26:53.518 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-09-17T09:26:53.518 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-09-17T09:26:53.519 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-17T09:26:53.520 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-09-17T09:26:53.520 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-09-17T09:26:53.520 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-09-17T09:26:53.520 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-09-17T09:26:53.520 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-09-17T09:26:53.520 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-09-17T09:26:53.520 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-09-17T09:26:53.520 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-09-17T09:26:53.520 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-09-17T09:26:53.521 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-09-17T09:26:53.521 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-09-17T09:26:53.521 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-09-17T09:26:53.521 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-09-17T09:26:53.521 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-09-17T09:26:53.521 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-09-17T09:26:53.521 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-09-17T09:26:53.521 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-09-17T09:26:53.521 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-09-17T09:26:53.521 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-09-17T09:26:53.521 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-09-17T09:26:53.521 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-17T09:26:53.521 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-09-17T09:26:53.522 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-09-17T09:26:53.523 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-09-17T09:26:53.523 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-09-17T09:26:53.523 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-09-17T09:26:53.523 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-09-17T09:26:53.523 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-09-17T09:26:53.523 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-09-17T09:26:53.523 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-09-17T09:26:53.523 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-09-17T09:26:53.523 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-09-17T09:26:53.523 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-09-17T09:26:53.523 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-09-17T09:26:53.523 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-09-17T09:26:53.523 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-09-17T09:26:53.524 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-09-17T09:26:53.525 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-17T09:26:53.525 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-17T09:26:53.525 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-17T09:26:53.525 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-17T09:26:53.525 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-09-17T09:26:53.525 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-09-17T09:26:53.525 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-09-17T09:26:53.525 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-09-17T09:26:53.525 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-17T09:26:53.525 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-09-17T09:26:53.525 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-09-17T09:26:53.525 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-09-17T09:26:53.526 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-09-17T09:26:53.527 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-09-17T09:26:53.528 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-09-17T09:26:53.528 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-09-17T09:26:53.528 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-09-17T09:26:53.528 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-09-17T09:26:53.528 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 bash[44043]: e361816f7c1489fb4a6d6bd4714b3bacde0cca491f2057463ea954ac85f4459e 2024-09-17T09:26:53.528 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-09-17T09:26:53.528 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-09-17T09:26:53.528 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-09-17T09:26:53.528 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-09-17T09:26:53.528 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-09-17T09:26:53.528 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 systemd[1]: Started Ceph grafana.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:26:53.528 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-09-17T09:26:53.528 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing HTTPServer" logger=server 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing PluginManager" logger=server 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Starting plugin search" logger=plugins 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing HooksService" logger=server 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing RemoteCache" logger=server 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing RenderingService" logger=server 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing AlertEngine" logger=server 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing QuotaService" logger=server 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing ServerLockService" logger=server 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2024-09-17T09:26:53.529 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2024-09-17T09:26:53.530 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing LoginService" logger=server 2024-09-17T09:26:53.530 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing SearchService" logger=server 2024-09-17T09:26:53.530 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing TracingService" logger=server 2024-09-17T09:26:53.530 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2024-09-17T09:26:53.530 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing CleanUpService" logger=server 2024-09-17T09:26:53.530 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing NotificationService" logger=server 2024-09-17T09:26:53.530 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2024-09-17T09:26:53.530 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2024-09-17T09:26:53.530 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki 2024-09-17T09:26:53.530 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2024-09-17T09:26:53.530 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2024-09-17T09:26:53.531 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="Initializing Stream Manager" 2024-09-17T09:26:53.531 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:26:53+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-09-17T09:26:53.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:52.612316+0000 mgr.x (mgr.34775) 66 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:53.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.420545+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:53.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.427209+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:53.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.434265+0000 mon.b (mon.2) 62 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:26:53.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.435486+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T09:26:53.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.442852+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:53.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.451315+0000 mon.b (mon.2) 64 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:26:53.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.456758+0000 mon.b (mon.2) 65 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:26:53.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.460812+0000 mon.b (mon.2) 66 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:26:53.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.461870+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T09:26:53.982 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.466060+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:53.982 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.474672+0000 mon.b (mon.2) 68 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:26:53.982 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.476101+0000 mon.b (mon.2) 69 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T09:26:53.982 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.480062+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:53.982 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.525974+0000 mon.b (mon.2) 70 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:26:53.982 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.530239+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:53.982 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:52.612316+0000 mgr.x (mgr.34775) 66 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:53.982 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.420545+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:53.982 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.427209+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:53.983 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.434265+0000 mon.b (mon.2) 62 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:26:53.983 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.435486+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T09:26:53.983 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.442852+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:53.983 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.451315+0000 mon.b (mon.2) 64 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:26:53.983 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.456758+0000 mon.b (mon.2) 65 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:26:53.983 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.460812+0000 mon.b (mon.2) 66 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:26:53.983 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.461870+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T09:26:53.983 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.466060+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:53.983 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.474672+0000 mon.b (mon.2) 68 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:26:53.983 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.476101+0000 mon.b (mon.2) 69 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T09:26:53.983 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.480062+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:53.983 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.525974+0000 mon.b (mon.2) 70 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:26:53.983 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.530239+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:54.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:52.612316+0000 mgr.x (mgr.34775) 66 : cluster [DBG] pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:54.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.420545+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:54.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.427209+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:54.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.434265+0000 mon.b (mon.2) 62 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:26:54.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.435486+0000 mon.b (mon.2) 63 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T09:26:54.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.442852+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:54.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.451315+0000 mon.b (mon.2) 64 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:26:54.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.456758+0000 mon.b (mon.2) 65 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:26:54.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.460812+0000 mon.b (mon.2) 66 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:26:54.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.461870+0000 mon.b (mon.2) 67 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T09:26:54.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.466060+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:54.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.474672+0000 mon.b (mon.2) 68 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:26:54.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.476101+0000 mon.b (mon.2) 69 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T09:26:54.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.480062+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:54.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.525974+0000 mon.b (mon.2) 70 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:26:54.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.530239+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:55.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.434607+0000 mgr.x (mgr.34775) 67 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:26:55.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.436101+0000 mgr.x (mgr.34775) 68 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T09:26:55.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.451836+0000 mgr.x (mgr.34775) 69 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:26:55.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.457109+0000 mgr.x (mgr.34775) 70 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:26:55.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.461139+0000 mgr.x (mgr.34775) 71 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:26:55.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.462151+0000 mgr.x (mgr.34775) 72 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T09:26:55.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.475024+0000 mgr.x (mgr.34775) 73 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:26:55.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:53.476360+0000 mgr.x (mgr.34775) 74 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T09:26:55.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:53.531561+0000 mgr.x (mgr.34775) 75 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T09:26:55.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:53.532060+0000 mgr.x (mgr.34775) 76 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T09:26:55.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.434607+0000 mgr.x (mgr.34775) 67 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:26:55.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.436101+0000 mgr.x (mgr.34775) 68 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T09:26:55.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.451836+0000 mgr.x (mgr.34775) 69 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:26:55.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.457109+0000 mgr.x (mgr.34775) 70 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:26:55.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.461139+0000 mgr.x (mgr.34775) 71 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:26:55.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.462151+0000 mgr.x (mgr.34775) 72 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T09:26:55.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.475024+0000 mgr.x (mgr.34775) 73 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:26:55.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:53.476360+0000 mgr.x (mgr.34775) 74 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T09:26:55.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:53.531561+0000 mgr.x (mgr.34775) 75 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T09:26:55.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:53.532060+0000 mgr.x (mgr.34775) 76 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T09:26:55.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.434607+0000 mgr.x (mgr.34775) 67 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:26:55.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.436101+0000 mgr.x (mgr.34775) 68 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T09:26:55.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.451836+0000 mgr.x (mgr.34775) 69 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:26:55.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.457109+0000 mgr.x (mgr.34775) 70 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:26:55.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.461139+0000 mgr.x (mgr.34775) 71 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:26:55.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.462151+0000 mgr.x (mgr.34775) 72 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi103.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T09:26:55.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.475024+0000 mgr.x (mgr.34775) 73 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:26:55.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:53.476360+0000 mgr.x (mgr.34775) 74 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi103.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T09:26:55.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:53.531561+0000 mgr.x (mgr.34775) 75 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T09:26:55.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:53.532060+0000 mgr.x (mgr.34775) 76 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T09:26:56.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:54.613273+0000 mgr.x (mgr.34775) 77 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:56.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:55.002943+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.29:0/1178499792' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:26:56.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:55.338635+0000 mon.a (mon.0) 726 : audit [INF] from='client.? 172.21.15.29:0/241538653' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: access denied 2024-09-17T09:26:56.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:55.504600+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:56.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:55.506708+0000 mon.b (mon.2) 71 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:26:56.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:55.507007+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:26:56.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:55.507905+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:26:56.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:26:55.508705+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:26:56.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:54.613273+0000 mgr.x (mgr.34775) 77 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:56.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:55.002943+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.29:0/1178499792' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:26:56.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:55.338635+0000 mon.a (mon.0) 726 : audit [INF] from='client.? 172.21.15.29:0/241538653' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: access denied 2024-09-17T09:26:56.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:55.504600+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:56.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:55.506708+0000 mon.b (mon.2) 71 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:26:56.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:55.507007+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:26:56.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:55.507905+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:26:56.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:26:55.508705+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:26:56.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:54.613273+0000 mgr.x (mgr.34775) 77 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:56.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:55.002943+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.29:0/1178499792' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:26:56.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:55.338635+0000 mon.a (mon.0) 726 : audit [INF] from='client.? 172.21.15.29:0/241538653' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: access denied 2024-09-17T09:26:56.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:55.504600+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:26:56.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:55.506708+0000 mon.b (mon.2) 71 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:26:56.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:55.507007+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:26:56.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:55.507905+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:26:56.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:26:55.508705+0000 mon.b (mon.2) 73 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:26:56.939 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:55.500009+0000 mgr.x (mgr.34775) 78 : cephadm [INF] Upgrade: Updating mgr.y 2024-09-17T09:26:56.939 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:26:55.509576+0000 mgr.x (mgr.34775) 79 : cephadm [INF] Deploying daemon mgr.y on smithi029 2024-09-17T09:26:56.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:55.500009+0000 mgr.x (mgr.34775) 78 : cephadm [INF] Upgrade: Updating mgr.y 2024-09-17T09:26:56.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:26:55.509576+0000 mgr.x (mgr.34775) 79 : cephadm [INF] Deploying daemon mgr.y on smithi029 2024-09-17T09:26:57.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:55.500009+0000 mgr.x (mgr.34775) 78 : cephadm [INF] Upgrade: Updating mgr.y 2024-09-17T09:26:57.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:26:55.509576+0000 mgr.x (mgr.34775) 79 : cephadm [INF] Deploying daemon mgr.y on smithi029 2024-09-17T09:26:57.905 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:56.613696+0000 mgr.x (mgr.34775) 80 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:57.905 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:56.613696+0000 mgr.x (mgr.34775) 80 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:58.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:56.613696+0000 mgr.x (mgr.34775) 80 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:26:58.542 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:26:58 smithi029 systemd[1]: Stopping Ceph mgr.y for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:26:58.798 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:26:58 smithi029 podman[53100]: 2024-09-17 09:26:58.743336552 +0000 UTC m=+0.160900255 container died 9d130c67b36dadbd4e03fdaf106266cf5fd868b8f96d8dfd5ce819a4552171a2 (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, RELEASE=HEAD, org.label-schema.build-date=20201204, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, maintainer=Dimitri Savineau , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-09-17T09:26:59.279 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:26:59 smithi029 podman[53100]: 2024-09-17 09:26:59.1990188 +0000 UTC m=+0.616582548 container cleanup 9d130c67b36dadbd4e03fdaf106266cf5fd868b8f96d8dfd5ce819a4552171a2 (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=-16.2.0, RELEASE=HEAD) 2024-09-17T09:26:59.280 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:26:59 smithi029 bash[53100]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y 2024-09-17T09:26:59.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:26:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:26:58.614451+0000 mgr.x (mgr.34775) 81 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:59.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:26:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:26:58.614451+0000 mgr.x (mgr.34775) 81 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:26:59.548 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:26:59 smithi029 podman[53114]: 2024-09-17 09:26:59.462182158 +0000 UTC m=+0.722738383 container remove 9d130c67b36dadbd4e03fdaf106266cf5fd868b8f96d8dfd5ce819a4552171a2 (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.y, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2) 2024-09-17T09:26:59.548 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:26:59 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:26:59.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:26:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:26:58.614451+0000 mgr.x (mgr.34775) 81 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:00.298 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:26:59 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-09-17T09:27:00.298 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:26:59 smithi029 systemd[1]: Stopped Ceph mgr.y for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:00.298 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:00 smithi029 systemd[1]: Starting Ceph mgr.y for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:27:00.571 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 2. 2024-09-17T09:27:00.571 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:00.571 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:27:00.571 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 podman[44106]: 2024-09-17 09:27:00.341934739 +0000 UTC m=+0.017760187 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:27:00.571 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 podman[44106]: 2024-09-17 09:27:00.459386039 +0000 UTC m=+0.135211487 container create 3dd9177574f43854f6cc7b22bbd33f2aa10f0e2a47c6cf9759952481c3a0b7a0 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:00.573 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:00 smithi029 podman[53252]: 2024-09-17 09:27:00.224694289 +0000 UTC m=+0.016438331 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:27:00.573 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:00 smithi029 podman[53252]: 2024-09-17 09:27:00.348764143 +0000 UTC m=+0.140508177 container create 29f7e5322eb6da976f12a34a19f60fb356a7097563c3ca8fc67ef9181f7348d7 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:27:00.573 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:00 smithi029 podman[53252]: 2024-09-17 09:27:00.569279788 +0000 UTC m=+0.361023829 container init 29f7e5322eb6da976f12a34a19f60fb356a7097563c3ca8fc67ef9181f7348d7 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-17T09:27:00.573 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:00 smithi029 podman[53252]: 2024-09-17 09:27:00.572238958 +0000 UTC m=+0.363982994 container start 29f7e5322eb6da976f12a34a19f60fb356a7097563c3ca8fc67ef9181f7348d7 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-17T09:27:00.925 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:00 smithi029 ceph-mgr[53270]: -- 172.21.15.29:0/3719745945 <== mon.1 v2:172.21.15.29:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5570aa37ed00 con 0x5570aa353000 2024-09-17T09:27:00.926 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:00 smithi029 bash[53252]: 29f7e5322eb6da976f12a34a19f60fb356a7097563c3ca8fc67ef9181f7348d7 2024-09-17T09:27:00.926 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:00 smithi029 systemd[1]: Started Ceph mgr.y for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:00.926 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:00.695+0000 7f772a6a7040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:27:00.926 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:00.761+0000 7f772a6a7040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:27:00.926 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:00.923+0000 7f772a6a7040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:27:01.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 podman[44106]: 2024-09-17 09:27:00.694760699 +0000 UTC m=+0.370586149 container init 3dd9177574f43854f6cc7b22bbd33f2aa10f0e2a47c6cf9759952481c3a0b7a0 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:01.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 podman[44106]: 2024-09-17 09:27:00.697478986 +0000 UTC m=+0.373304435 container start 3dd9177574f43854f6cc7b22bbd33f2aa10f0e2a47c6cf9759952481c3a0b7a0 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:01.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.726Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:27:01.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.726Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:27:01.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.726Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:27:01.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.726Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:27:01.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.726Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:27:01.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.726Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:27:01.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.727Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:27:01.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.727Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:27:01.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.729Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:27:01.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.730Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-09-17T09:27:01.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.737Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-09-17T09:27:01.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.738Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-09-17T09:27:01.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.738Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-09-17T09:27:01.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.738Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-09-17T09:27:01.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.738Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=8.578046ms 2024-09-17T09:27:01.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.740Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:27:01.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.740Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:27:01.008 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.740Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:27:01.008 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.740Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:27:01.008 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.740Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:27:01.008 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.740Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:27:01.008 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.740Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:27:01.008 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.740Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:27:01.008 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.740Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:27:01.008 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.740Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:27:01.008 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.740Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:27:01.008 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.740Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:27:01.008 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=info ts=2024-09-17T09:27:00.740Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:27:01.008 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44121]: level=error ts=2024-09-17T09:27:00.741Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:27:01.009 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 bash[44106]: 3dd9177574f43854f6cc7b22bbd33f2aa10f0e2a47c6cf9759952481c3a0b7a0 2024-09-17T09:27:01.009 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:01.009 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:00 smithi103 podman[44138]: 2024-09-17 09:27:00.801167474 +0000 UTC m=+0.052683697 container died 3dd9177574f43854f6cc7b22bbd33f2aa10f0e2a47c6cf9759952481c3a0b7a0 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:01.298 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:01.002+0000 7f772a6a7040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:27:01.298 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:01.168+0000 7f772a6a7040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:27:01.300 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:27:01.682 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:01.335+0000 7f772a6a7040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:27:01.682 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:01.679+0000 7f772a6a7040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:27:01.954 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:01.750+0000 7f772a6a7040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:27:01.954 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:01.951+0000 7f772a6a7040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:27:01.954 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:00.614838+0000 mgr.x (mgr.34775) 82 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:01.955 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:00.744359+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:01.955 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:00.750742+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:01.955 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:01.010451+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:01.955 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:01.016891+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:01.955 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:00.614838+0000 mgr.x (mgr.34775) 82 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:01.955 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:00.744359+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:01.955 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:00.750742+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:01.955 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:01.010451+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:01.955 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:01.016891+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:02.000 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:27:02.001 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 starting - - - - 2024-09-17T09:27:02.001 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 starting - - - - 2024-09-17T09:27:02.001 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 starting - - - - 2024-09-17T09:27:02.001 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443 running (75s) 63s ago 9m 459M - 18.2.4-855-g19e2a2be 980a0bfcd684 d35594bdef28 2024-09-17T09:27:02.001 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 starting - - - - 2024-09-17T09:27:02.001 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (10m) 56s ago 10m 50.2M 2048M 16.2.0 24ecd6d5f14c f3476652dfdd 2024-09-17T09:27:02.001 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (9m) 63s ago 9m 39.5M 2048M 16.2.0 24ecd6d5f14c 203d1f26d7b8 2024-09-17T09:27:02.001 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (9m) 56s ago 9m 50.1M 2048M 16.2.0 24ecd6d5f14c aa05f3e5656a 2024-09-17T09:27:02.001 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 starting - - - - 2024-09-17T09:27:02.001 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 starting - - - - 2024-09-17T09:27:02.001 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (8m) 56s ago 8m 44.8M 4096M 16.2.0 24ecd6d5f14c 991731ce08b5 2024-09-17T09:27:02.001 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (8m) 56s ago 8m 44.9M 4096M 16.2.0 24ecd6d5f14c d5bce63fadc8 2024-09-17T09:27:02.002 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (8m) 56s ago 8m 41.5M 4096M 16.2.0 24ecd6d5f14c e9678d2f1685 2024-09-17T09:27:02.002 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (7m) 56s ago 7m 48.3M 4096M 16.2.0 24ecd6d5f14c dcf7122a7d99 2024-09-17T09:27:02.002 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (7m) 63s ago 7m 47.7M 4096M 16.2.0 24ecd6d5f14c 048c13025d32 2024-09-17T09:27:02.002 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (7m) 63s ago 7m 46.3M 4096M 16.2.0 24ecd6d5f14c 10a64919fbc9 2024-09-17T09:27:02.002 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (7m) 63s ago 7m 43.0M 4096M 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:27:02.002 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (6m) 63s ago 6m 48.7M 4096M 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:27:02.002 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 starting - - - - 2024-09-17T09:27:02.002 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (4m) 56s ago 4m 38.9M - 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:27:02.002 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (4m) 63s ago 4m 38.8M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:27:02.002 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (4m) 56s ago 4m 36.4M - 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:27:02.002 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (4m) 63s ago 4m 36.0M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:27:02.005 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:01 smithi103 podman[44138]: 2024-09-17 09:27:01.642674068 +0000 UTC m=+0.894190289 container remove 3dd9177574f43854f6cc7b22bbd33f2aa10f0e2a47c6cf9759952481c3a0b7a0 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:02.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:01 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:27:02.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:01 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:27:02.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:00.614838+0000 mgr.x (mgr.34775) 82 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:02.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:00.744359+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:02.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:00.750742+0000 mon.a (mon.0) 730 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:02.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:01.010451+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:02.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:01.016891+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:02.448 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:27:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:27:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2024-09-17T09:27:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:27:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:27:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2024-09-17T09:27:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T09:27:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:27:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:27:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T09:27:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:27:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": {}, 2024-09-17T09:27:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:27:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:27:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:27:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:27:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 16, 2024-09-17T09:27:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T09:27:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:27:02.450 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:27:02.806 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:27:02.806 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:27:02.806 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:27:02.806 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:27:02.806 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [], 2024-09-17T09:27:02.806 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "", 2024-09-17T09:27:02.807 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Currently upgrading mgr daemons", 2024-09-17T09:27:02.807 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:27:02.807 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:27:03.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:01.283974+0000 mgr.x (mgr.34775) 83 : audit [DBG] from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:03.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:01.644235+0000 mgr.x (mgr.34775) 84 : audit [DBG] from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:03.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:02.447721+0000 mon.a (mon.0) 733 : audit [DBG] from='client.? 172.21.15.29:0/2962848495' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:27:03.017 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:01.283974+0000 mgr.x (mgr.34775) 83 : audit [DBG] from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:03.017 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:01.644235+0000 mgr.x (mgr.34775) 84 : audit [DBG] from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:03.017 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:02.447721+0000 mon.a (mon.0) 733 : audit [DBG] from='client.? 172.21.15.29:0/2962848495' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:27:03.018 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:01.283974+0000 mgr.x (mgr.34775) 83 : audit [DBG] from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:03.018 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:01.644235+0000 mgr.x (mgr.34775) 84 : audit [DBG] from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:03.018 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:02.447721+0000 mon.a (mon.0) 733 : audit [DBG] from='client.? 172.21.15.29:0/2962848495' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:27:03.018 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:02.929+0000 7f772a6a7040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:27:03.019 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:03.014+0000 7f772a6a7040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:27:03.255 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T09:27:03.256 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T09:27:03.256 INFO:teuthology.orchestra.run.smithi029.stdout: daemon iscsi.foo.smithi029.gellkh on smithi029 is in error state 2024-09-17T09:27:04.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:01.992591+0000 mgr.x (mgr.34775) 85 : audit [DBG] from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:04.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:02.615150+0000 mgr.x (mgr.34775) 86 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:04.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:02.805678+0000 mgr.x (mgr.34775) 87 : audit [DBG] from='client.34967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:04.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:03.255448+0000 mon.a (mon.0) 734 : audit [DBG] from='client.? 172.21.15.29:0/2365796249' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:27:04.018 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:01.992591+0000 mgr.x (mgr.34775) 85 : audit [DBG] from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:04.018 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:02.615150+0000 mgr.x (mgr.34775) 86 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:04.018 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:02.805678+0000 mgr.x (mgr.34775) 87 : audit [DBG] from='client.34967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:04.018 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:03.255448+0000 mon.a (mon.0) 734 : audit [DBG] from='client.? 172.21.15.29:0/2365796249' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:27:04.018 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:01.992591+0000 mgr.x (mgr.34775) 85 : audit [DBG] from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:04.018 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:02.615150+0000 mgr.x (mgr.34775) 86 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:04.018 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:02.805678+0000 mgr.x (mgr.34775) 87 : audit [DBG] from='client.34967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:04.018 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:03.255448+0000 mon.a (mon.0) 734 : audit [DBG] from='client.? 172.21.15.29:0/2365796249' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:27:04.019 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:03.866+0000 7f772a6a7040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:27:04.019 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:04.015+0000 7f772a6a7040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:27:04.272 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:04.170+0000 7f772a6a7040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:27:04.272 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:04.234+0000 7f772a6a7040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:27:04.530 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:04.391+0000 7f772a6a7040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:27:04.530 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:04.464+0000 7f772a6a7040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:27:04.530 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:04.527+0000 7f772a6a7040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:27:04.798 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:04.691+0000 7f772a6a7040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:27:04.798 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:04.765+0000 7f772a6a7040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:27:05.135 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:04.838+0000 7f772a6a7040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:27:05.135 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:04.938+0000 7f772a6a7040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:27:05.135 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:05.000+0000 7f772a6a7040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:27:05.135 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:05.061+0000 7f772a6a7040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:27:06.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:04.615773+0000 mgr.x (mgr.34775) 88 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:06.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:04.615773+0000 mgr.x (mgr.34775) 88 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:06.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:04.615773+0000 mgr.x (mgr.34775) 88 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:06.048 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:05.909+0000 7f772a6a7040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:27:07.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:05.913380+0000 mon.a (mon.0) 735 : cluster [DBG] Standby manager daemon y restarted 2024-09-17T09:27:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:05.913527+0000 mon.a (mon.0) 736 : cluster [DBG] Standby manager daemon y started 2024-09-17T09:27:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:05.914369+0000 mon.c (mon.1) 36 : audit [DBG] from='mgr.? 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:27:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:05.914821+0000 mon.c (mon.1) 37 : audit [DBG] from='mgr.? 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:27:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:05.915628+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.? 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T09:27:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:05.916031+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.? 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:27:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:06.165318+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:06.170864+0000 mon.a (mon.0) 738 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:06.206843+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:06.212508+0000 mon.a (mon.0) 740 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:06.726506+0000 mon.a (mon.0) 741 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:06.732465+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:05.913380+0000 mon.a (mon.0) 735 : cluster [DBG] Standby manager daemon y restarted 2024-09-17T09:27:07.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:05.913527+0000 mon.a (mon.0) 736 : cluster [DBG] Standby manager daemon y started 2024-09-17T09:27:07.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:05.914369+0000 mon.c (mon.1) 36 : audit [DBG] from='mgr.? 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:27:07.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:05.914821+0000 mon.c (mon.1) 37 : audit [DBG] from='mgr.? 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:27:07.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:05.915628+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.? 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T09:27:07.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:05.916031+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.? 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:27:07.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:06.165318+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:06.170864+0000 mon.a (mon.0) 738 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:06.206843+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:06.212508+0000 mon.a (mon.0) 740 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:06.726506+0000 mon.a (mon.0) 741 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:06.732465+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.033 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:05.913380+0000 mon.a (mon.0) 735 : cluster [DBG] Standby manager daemon y restarted 2024-09-17T09:27:07.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:05.913527+0000 mon.a (mon.0) 736 : cluster [DBG] Standby manager daemon y started 2024-09-17T09:27:07.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:05.914369+0000 mon.c (mon.1) 36 : audit [DBG] from='mgr.? 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T09:27:07.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:05.914821+0000 mon.c (mon.1) 37 : audit [DBG] from='mgr.? 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:27:07.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:05.915628+0000 mon.c (mon.1) 38 : audit [DBG] from='mgr.? 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T09:27:07.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:05.916031+0000 mon.c (mon.1) 39 : audit [DBG] from='mgr.? 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:27:07.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:06.165318+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:06.170864+0000 mon.a (mon.0) 738 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:06.206843+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:06.212508+0000 mon.a (mon.0) 740 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:06.726506+0000 mon.a (mon.0) 741 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:06.732465+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:07.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:06.616059+0000 mgr.x (mgr.34775) 89 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:07.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:06.764789+0000 mon.a (mon.0) 743 : cluster [DBG] mgrmap e20: x(active, since 74s), standbys: y 2024-09-17T09:27:07.871 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:06.616059+0000 mgr.x (mgr.34775) 89 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:07.871 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:06.764789+0000 mon.a (mon.0) 743 : cluster [DBG] mgrmap e20: x(active, since 74s), standbys: y 2024-09-17T09:27:08.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:06.616059+0000 mgr.x (mgr.34775) 89 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:08.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:06.764789+0000 mon.a (mon.0) 743 : cluster [DBG] mgrmap e20: x(active, since 74s), standbys: y 2024-09-17T09:27:09.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:08.616844+0000 mgr.x (mgr.34775) 90 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T09:27:09.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:08.688331+0000 mon.a (mon.0) 744 : audit [DBG] from='client.? 172.21.15.29:0/1736913043' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:09.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:09.026441+0000 mon.a (mon.0) 745 : audit [INF] from='client.? 172.21.15.29:0/2315169873' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: access denied 2024-09-17T09:27:09.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:08.616844+0000 mgr.x (mgr.34775) 90 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T09:27:09.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:08.688331+0000 mon.a (mon.0) 744 : audit [DBG] from='client.? 172.21.15.29:0/1736913043' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:09.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:09.026441+0000 mon.a (mon.0) 745 : audit [INF] from='client.? 172.21.15.29:0/2315169873' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: access denied 2024-09-17T09:27:09.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:08.616844+0000 mgr.x (mgr.34775) 90 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T09:27:09.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:08.688331+0000 mon.a (mon.0) 744 : audit [DBG] from='client.? 172.21.15.29:0/1736913043' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:09.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:09.026441+0000 mon.a (mon.0) 745 : audit [INF] from='client.? 172.21.15.29:0/2315169873' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: access denied 2024-09-17T09:27:11.944 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:10.617549+0000 mgr.x (mgr.34775) 91 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:11.945 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:11 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 3. 2024-09-17T09:27:11.945 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:11 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:11.945 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:11 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:27:12.024 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:10.617549+0000 mgr.x (mgr.34775) 91 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:12.024 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:10.617549+0000 mgr.x (mgr.34775) 91 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:12.223 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:11 smithi103 podman[44936]: 2024-09-17 09:27:11.847539613 +0000 UTC m=+0.018342546 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:27:12.223 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:11 smithi103 podman[44936]: 2024-09-17 09:27:11.969664001 +0000 UTC m=+0.140466953 container create 07afdf95800210b356b0d52613fdb1f5280a8d080e15164d0fdebe494503d105 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:12.223 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 podman[44936]: 2024-09-17 09:27:12.188551892 +0000 UTC m=+0.359354818 container init 07afdf95800210b356b0d52613fdb1f5280a8d080e15164d0fdebe494503d105 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:12.223 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 podman[44936]: 2024-09-17 09:27:12.191491637 +0000 UTC m=+0.362294547 container start 07afdf95800210b356b0d52613fdb1f5280a8d080e15164d0fdebe494503d105 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:12.223 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.221Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:27:12.506 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.221Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:27:12.506 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.221Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:27:12.506 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.221Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:27:12.506 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.221Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.221Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.223Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.223Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.226Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.226Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.233Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.233Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.233Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.233Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.233Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.233Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=7.10105ms 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.235Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.235Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:27:12.507 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.235Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:27:12.508 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.235Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:27:12.508 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.235Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:27:12.508 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.235Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:27:12.508 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.235Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:27:12.508 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.235Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:27:12.508 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.235Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:27:12.508 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.235Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:27:12.508 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.236Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:27:12.508 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.236Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:27:12.508 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=info ts=2024-09-17T09:27:12.236Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:27:12.508 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[44951]: level=error ts=2024-09-17T09:27:12.236Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:27:12.508 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 bash[44936]: 07afdf95800210b356b0d52613fdb1f5280a8d080e15164d0fdebe494503d105 2024-09-17T09:27:12.508 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:12.509 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 podman[44969]: 2024-09-17 09:27:12.344670264 +0000 UTC m=+0.096866812 container died 07afdf95800210b356b0d52613fdb1f5280a8d080e15164d0fdebe494503d105 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:13.255 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 podman[44969]: 2024-09-17 09:27:12.951512147 +0000 UTC m=+0.703708681 container remove 07afdf95800210b356b0d52613fdb1f5280a8d080e15164d0fdebe494503d105 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:13.255 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:27:13.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:12 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:27:14.217 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:13.870+0000 7fae0b9d1640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-17T09:27:14.217 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: ignoring --setuser ceph since I am not root 2024-09-17T09:27:14.217 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: ignoring --setgroup ceph since I am not root 2024-09-17T09:27:14.217 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:13.918+0000 7f71aea04640 1 -- 172.21.15.103:0/2288929540 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562863156d00 con 0x56286312b400 2024-09-17T09:27:14.217 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:14.009+0000 7f71b125f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:27:14.217 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:14.065+0000 7f71b125f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:27:14.218 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:14.214+0000 7f71b125f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:27:14.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:12.617984+0000 mgr.x (mgr.34775) 92 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:14.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:12.855926+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:14.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:12.862233+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:14.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:12.863615+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:14.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:12.864388+0000 mon.b (mon.2) 75 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:27:14.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:12.864953+0000 mgr.x (mgr.34775) 93 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:27:14.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:12.865397+0000 mgr.x (mgr.34775) 94 : cluster [DBG] pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2024-09-17T09:27:14.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:12.865589+0000 mgr.x (mgr.34775) 95 : cluster [DBG] pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 124 B/s rd, 0 op/s 2024-09-17T09:27:14.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:12.869116+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:14.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.007587+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:27:14.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.012162+0000 mon.b (mon.2) 77 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T09:27:14.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.012477+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T09:27:14.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:13.016589+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-17T09:27:14.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:12.617984+0000 mgr.x (mgr.34775) 92 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:14.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:12.855926+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:12.862233+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:12.863615+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:12.864388+0000 mon.b (mon.2) 75 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:27:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:12.864953+0000 mgr.x (mgr.34775) 93 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:27:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:12.865397+0000 mgr.x (mgr.34775) 94 : cluster [DBG] pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2024-09-17T09:27:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:12.865589+0000 mgr.x (mgr.34775) 95 : cluster [DBG] pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 124 B/s rd, 0 op/s 2024-09-17T09:27:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:12.869116+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:14.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.007587+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:27:14.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.012162+0000 mon.b (mon.2) 77 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T09:27:14.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.012477+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T09:27:14.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:13.016589+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-17T09:27:14.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:12.617984+0000 mgr.x (mgr.34775) 92 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:12.855926+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:12.862233+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:12.863615+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:12.864388+0000 mon.b (mon.2) 75 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:27:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:12.864953+0000 mgr.x (mgr.34775) 93 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:27:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:12.865397+0000 mgr.x (mgr.34775) 94 : cluster [DBG] pgmap v47: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 99 B/s rd, 0 op/s 2024-09-17T09:27:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:12.865589+0000 mgr.x (mgr.34775) 95 : cluster [DBG] pgmap v48: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 124 B/s rd, 0 op/s 2024-09-17T09:27:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:12.869116+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.34775 ' entity='mgr.x' 2024-09-17T09:27:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.007587+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:27:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.012162+0000 mon.b (mon.2) 77 : audit [INF] from='mgr.34775 172.21.15.103:0/2112299339' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T09:27:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.012477+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T09:27:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:13.016589+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-17T09:27:14.472 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:14.281+0000 7f71b125f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:27:14.472 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:14.427+0000 7f71b125f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:27:14.755 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:14.579+0000 7f71b125f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:27:15.173 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:13.860169+0000 mon.a (mon.0) 751 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:27:15.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.873765+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T09:27:15.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:13.873880+0000 mon.a (mon.0) 753 : cluster [DBG] mgrmap e21: y(active, starting, since 0.860728s) 2024-09-17T09:27:15.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:13.875284+0000 mon.a (mon.0) 754 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:27:15.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.875957+0000 mon.c (mon.1) 40 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:27:15.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.876154+0000 mon.c (mon.1) 41 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:27:15.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.876364+0000 mon.c (mon.1) 42 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:27:15.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.876963+0000 mon.c (mon.1) 43 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:27:15.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.877204+0000 mon.c (mon.1) 44 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:27:15.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.877553+0000 mon.c (mon.1) 45 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:27:15.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.877893+0000 mon.c (mon.1) 46 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:27:15.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.878197+0000 mon.c (mon.1) 47 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:27:15.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.878513+0000 mon.c (mon.1) 48 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:27:15.175 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.878855+0000 mon.c (mon.1) 49 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:27:15.175 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.879223+0000 mon.c (mon.1) 50 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:27:15.175 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.879484+0000 mon.c (mon.1) 51 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:27:15.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:13.860169+0000 mon.a (mon.0) 751 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:27:15.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.873765+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T09:27:15.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:13.873880+0000 mon.a (mon.0) 753 : cluster [DBG] mgrmap e21: y(active, starting, since 0.860728s) 2024-09-17T09:27:15.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:13.875284+0000 mon.a (mon.0) 754 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:27:15.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.875957+0000 mon.c (mon.1) 40 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:27:15.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.876154+0000 mon.c (mon.1) 41 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:27:15.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.876364+0000 mon.c (mon.1) 42 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:27:15.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.876963+0000 mon.c (mon.1) 43 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:27:15.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.877204+0000 mon.c (mon.1) 44 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:27:15.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.877553+0000 mon.c (mon.1) 45 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:27:15.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.877893+0000 mon.c (mon.1) 46 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:27:15.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.878197+0000 mon.c (mon.1) 47 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:27:15.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.878513+0000 mon.c (mon.1) 48 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:27:15.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.878855+0000 mon.c (mon.1) 49 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:27:15.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.879223+0000 mon.c (mon.1) 50 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:27:15.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.879484+0000 mon.c (mon.1) 51 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:27:15.176 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.879828+0000 mon.c (mon.1) 52 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:27:15.177 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.880050+0000 mon.c (mon.1) 53 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:27:15.177 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:13.880542+0000 mon.c (mon.1) 54 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:27:15.177 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:14.124580+0000 mon.a (mon.0) 755 : cluster [INF] Manager daemon y is now available 2024-09-17T09:27:15.177 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:14.135989+0000 mgr.y (mgr.25054) 1 : cephadm [INF] inventory: adjusted host smithi103 addr 'smithi103' -> '172.21.15.103' 2024-09-17T09:27:15.177 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:14.139702+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:15.177 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:14.178324+0000 mon.c (mon.1) 55 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:27:15.177 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:14.179172+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:27:15.177 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:14.190587+0000 mon.c (mon.1) 56 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:27:15.177 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:14.191268+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:27:15.177 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.879828+0000 mon.c (mon.1) 52 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:27:15.177 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.880050+0000 mon.c (mon.1) 53 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:27:15.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:13.880542+0000 mon.c (mon.1) 54 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:27:15.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:14.124580+0000 mon.a (mon.0) 755 : cluster [INF] Manager daemon y is now available 2024-09-17T09:27:15.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:14.135989+0000 mgr.y (mgr.25054) 1 : cephadm [INF] inventory: adjusted host smithi103 addr 'smithi103' -> '172.21.15.103' 2024-09-17T09:27:15.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:14.139702+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:15.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:14.178324+0000 mon.c (mon.1) 55 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:27:15.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:14.179172+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:27:15.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:14.190587+0000 mon.c (mon.1) 56 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:27:15.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:14 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:14.191268+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:27:15.193 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:14.937+0000 7f71b125f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:27:15.193 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:15.006+0000 7f71b125f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:27:15.193 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:15.191+0000 7f71b125f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:27:15.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:13.860169+0000 mon.a (mon.0) 751 : cluster [WRN] Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:27:15.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.873765+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.34775 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T09:27:15.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:13.873880+0000 mon.a (mon.0) 753 : cluster [DBG] mgrmap e21: y(active, starting, since 0.860728s) 2024-09-17T09:27:15.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:13.875284+0000 mon.a (mon.0) 754 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:27:15.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.875957+0000 mon.c (mon.1) 40 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:27:15.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.876154+0000 mon.c (mon.1) 41 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:27:15.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.876364+0000 mon.c (mon.1) 42 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:27:15.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.876963+0000 mon.c (mon.1) 43 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:27:15.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.877204+0000 mon.c (mon.1) 44 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:27:15.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.877553+0000 mon.c (mon.1) 45 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:27:15.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.877893+0000 mon.c (mon.1) 46 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:27:15.194 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.878197+0000 mon.c (mon.1) 47 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:27:15.195 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.878513+0000 mon.c (mon.1) 48 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:27:15.195 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.878855+0000 mon.c (mon.1) 49 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:27:15.195 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.879223+0000 mon.c (mon.1) 50 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:27:15.195 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.879484+0000 mon.c (mon.1) 51 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:27:15.195 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.879828+0000 mon.c (mon.1) 52 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:27:15.195 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.880050+0000 mon.c (mon.1) 53 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:27:15.195 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:13.880542+0000 mon.c (mon.1) 54 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:27:15.195 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:14.124580+0000 mon.a (mon.0) 755 : cluster [INF] Manager daemon y is now available 2024-09-17T09:27:15.195 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:14.135989+0000 mgr.y (mgr.25054) 1 : cephadm [INF] inventory: adjusted host smithi103 addr 'smithi103' -> '172.21.15.103' 2024-09-17T09:27:15.195 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:14.139702+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:15.195 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:14.178324+0000 mon.c (mon.1) 55 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:27:15.195 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:14.179172+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:27:15.195 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:14.190587+0000 mon.c (mon.1) 56 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:27:15.196 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:14.191268+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:27:16.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:14.880143+0000 mon.a (mon.0) 759 : cluster [DBG] mgrmap e22: y(active, since 1.86699s), standbys: x 2024-09-17T09:27:16.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:14.890139+0000 mon.c (mon.1) 57 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:27:16.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:14.892254+0000 mgr.y (mgr.25054) 2 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:16.088 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:14.880143+0000 mon.a (mon.0) 759 : cluster [DBG] mgrmap e22: y(active, since 1.86699s), standbys: x 2024-09-17T09:27:16.088 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:14.890139+0000 mon.c (mon.1) 57 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:27:16.088 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:14.892254+0000 mgr.y (mgr.25054) 2 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:16.088 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:16.022+0000 7f768aaaa640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:09:27:16] ENGINE Error in HTTPServer.serve 2024-09-17T09:27:16.088 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: Traceback (most recent call last): 2024-09-17T09:27:16.088 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:27:16.088 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: self._connections.run(self.expiration_interval) 2024-09-17T09:27:16.089 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:27:16.089 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: self._run(expiration_interval) 2024-09-17T09:27:16.089 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:27:16.089 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:27:16.089 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:27:16.089 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:27:16.089 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:27:16.089 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: s = self.context.wrap_socket( 2024-09-17T09:27:16.089 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:27:16.089 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: return self.sslsocket_class._create( 2024-09-17T09:27:16.089 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:27:16.089 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: self.do_handshake() 2024-09-17T09:27:16.089 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:27:16.090 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: self._sslobj.do_handshake() 2024-09-17T09:27:16.090 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:27:16.090 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:16.182 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:16.106+0000 7f71b125f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:27:16.182 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:16.180+0000 7f71b125f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:27:16.182 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:14.880143+0000 mon.a (mon.0) 759 : cluster [DBG] mgrmap e22: y(active, since 1.86699s), standbys: x 2024-09-17T09:27:16.182 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:14.890139+0000 mon.c (mon.1) 57 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:27:16.182 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:14.892254+0000 mgr.y (mgr.25054) 2 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:17.197 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:15.801207+0000 mgr.y (mgr.25054) 3 : cephadm [INF] [17/Sep/2024:09:27:15] ENGINE Bus STARTING 2024-09-17T09:27:17.198 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:15.872689+0000 mon.a (mon.0) 760 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T09:27:17.198 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:15.872722+0000 mon.a (mon.0) 761 : cluster [INF] Cluster is now healthy 2024-09-17T09:27:17.198 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:15.878985+0000 mgr.y (mgr.25054) 4 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:17.198 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:15.903228+0000 mgr.y (mgr.25054) 5 : cephadm [INF] [17/Sep/2024:09:27:15] ENGINE Serving on http://172.21.15.29:8765 2024-09-17T09:27:17.198 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:16.022142+0000 mgr.y (mgr.25054) 6 : cephadm [INF] [17/Sep/2024:09:27:16] ENGINE Serving on https://172.21.15.29:7150 2024-09-17T09:27:17.198 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:16.022222+0000 mgr.y (mgr.25054) 7 : cephadm [INF] [17/Sep/2024:09:27:16] ENGINE Bus STARTED 2024-09-17T09:27:17.198 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:16.023420+0000 mgr.y (mgr.25054) 8 : cephadm [ERR] [17/Sep/2024:09:27:16] ENGINE Error in HTTPServer.serve 2024-09-17T09:27:17.198 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Traceback (most recent call last): 2024-09-17T09:27:17.198 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:27:17.198 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: self._connections.run(self.expiration_interval) 2024-09-17T09:27:17.198 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: self._run(expiration_interval) 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: s = self.context.wrap_socket( 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: return self.sslsocket_class._create( 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: self.do_handshake() 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: self._sslobj.do_handshake() 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:27:17.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:27:17.200 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:16.961+0000 7f71b125f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:27:17.200 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:17.094+0000 7f71b125f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:27:17.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:15.801207+0000 mgr.y (mgr.25054) 3 : cephadm [INF] [17/Sep/2024:09:27:15] ENGINE Bus STARTING 2024-09-17T09:27:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:15.872689+0000 mon.a (mon.0) 760 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T09:27:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:15.872722+0000 mon.a (mon.0) 761 : cluster [INF] Cluster is now healthy 2024-09-17T09:27:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:15.878985+0000 mgr.y (mgr.25054) 4 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:15.903228+0000 mgr.y (mgr.25054) 5 : cephadm [INF] [17/Sep/2024:09:27:15] ENGINE Serving on http://172.21.15.29:8765 2024-09-17T09:27:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:16.022142+0000 mgr.y (mgr.25054) 6 : cephadm [INF] [17/Sep/2024:09:27:16] ENGINE Serving on https://172.21.15.29:7150 2024-09-17T09:27:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:16.022222+0000 mgr.y (mgr.25054) 7 : cephadm [INF] [17/Sep/2024:09:27:16] ENGINE Bus STARTED 2024-09-17T09:27:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:16.023420+0000 mgr.y (mgr.25054) 8 : cephadm [ERR] [17/Sep/2024:09:27:16] ENGINE Error in HTTPServer.serve 2024-09-17T09:27:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: Traceback (most recent call last): 2024-09-17T09:27:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:27:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: self._connections.run(self.expiration_interval) 2024-09-17T09:27:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:27:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: self._run(expiration_interval) 2024-09-17T09:27:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:27:17.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:27:17.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:27:17.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:27:17.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:27:17.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: s = self.context.wrap_socket( 2024-09-17T09:27:17.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:27:17.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: return self.sslsocket_class._create( 2024-09-17T09:27:17.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:27:17.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: self.do_handshake() 2024-09-17T09:27:17.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:27:17.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: self._sslobj.do_handshake() 2024-09-17T09:27:17.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:27:17.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: 2024-09-17T09:27:17.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:15.801207+0000 mgr.y (mgr.25054) 3 : cephadm [INF] [17/Sep/2024:09:27:15] ENGINE Bus STARTING 2024-09-17T09:27:17.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:15.872689+0000 mon.a (mon.0) 760 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T09:27:17.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:15.872722+0000 mon.a (mon.0) 761 : cluster [INF] Cluster is now healthy 2024-09-17T09:27:17.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:15.878985+0000 mgr.y (mgr.25054) 4 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:17.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:15.903228+0000 mgr.y (mgr.25054) 5 : cephadm [INF] [17/Sep/2024:09:27:15] ENGINE Serving on http://172.21.15.29:8765 2024-09-17T09:27:17.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:16.022142+0000 mgr.y (mgr.25054) 6 : cephadm [INF] [17/Sep/2024:09:27:16] ENGINE Serving on https://172.21.15.29:7150 2024-09-17T09:27:17.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:16.022222+0000 mgr.y (mgr.25054) 7 : cephadm [INF] [17/Sep/2024:09:27:16] ENGINE Bus STARTED 2024-09-17T09:27:17.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:16.023420+0000 mgr.y (mgr.25054) 8 : cephadm [ERR] [17/Sep/2024:09:27:16] ENGINE Error in HTTPServer.serve 2024-09-17T09:27:17.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Traceback (most recent call last): 2024-09-17T09:27:17.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:27:17.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: self._connections.run(self.expiration_interval) 2024-09-17T09:27:17.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:27:17.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: self._run(expiration_interval) 2024-09-17T09:27:17.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:27:17.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:27:17.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:27:17.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:27:17.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:27:17.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: s = self.context.wrap_socket( 2024-09-17T09:27:17.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:27:17.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: return self.sslsocket_class._create( 2024-09-17T09:27:17.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:27:17.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: self.do_handshake() 2024-09-17T09:27:17.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:27:17.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: self._sslobj.do_handshake() 2024-09-17T09:27:17.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:27:17.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:27:17.500 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:17.253+0000 7f71b125f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:27:17.500 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:17.333+0000 7f71b125f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:27:17.500 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:17.498+0000 7f71b125f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:27:17.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:17.577+0000 7f71b125f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:27:17.757 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:17.647+0000 7f71b125f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:27:18.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:16.889939+0000 mon.a (mon.0) 762 : cluster [DBG] mgrmap e23: y(active, since 3s), standbys: x 2024-09-17T09:27:18.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:16.889939+0000 mon.a (mon.0) 762 : cluster [DBG] mgrmap e23: y(active, since 3s), standbys: x 2024-09-17T09:27:18.092 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:16.889939+0000 mon.a (mon.0) 762 : cluster [DBG] mgrmap e23: y(active, since 3s), standbys: x 2024-09-17T09:27:18.093 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:17.805+0000 7f71b125f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:27:18.093 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:17.870+0000 7f71b125f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:27:18.093 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:17.947+0000 7f71b125f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:27:18.093 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:18.029+0000 7f71b125f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:27:18.093 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:18.090+0000 7f71b125f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:27:18.505 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:18.150+0000 7f71b125f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:27:19.255 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x[40906]: debug 2024-09-17T09:27:18.974+0000 7f71b125f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:27:19.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:17.879538+0000 mgr.y (mgr.25054) 9 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:19.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:18.895670+0000 mon.a (mon.0) 763 : cluster [DBG] mgrmap e24: y(active, since 5s), standbys: x 2024-09-17T09:27:19.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:18.978033+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.? 172.21.15.103:0/567347881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:27:19.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:18.978280+0000 mon.a (mon.0) 764 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T09:27:19.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:18.978398+0000 mon.a (mon.0) 765 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:27:19.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:18.979995+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.? 172.21.15.103:0/567347881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:27:19.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:18.980675+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.? 172.21.15.103:0/567347881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:27:19.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:18.980935+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.? 172.21.15.103:0/567347881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:27:19.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:17.879538+0000 mgr.y (mgr.25054) 9 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:19.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:18.895670+0000 mon.a (mon.0) 763 : cluster [DBG] mgrmap e24: y(active, since 5s), standbys: x 2024-09-17T09:27:19.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:18.978033+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.? 172.21.15.103:0/567347881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:27:19.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:18.978280+0000 mon.a (mon.0) 764 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T09:27:19.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:18.978398+0000 mon.a (mon.0) 765 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:27:19.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:18.979995+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.? 172.21.15.103:0/567347881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:27:19.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:18.980675+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.? 172.21.15.103:0/567347881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:27:19.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:18.980935+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.? 172.21.15.103:0/567347881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:27:19.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:17.879538+0000 mgr.y (mgr.25054) 9 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:19.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:18.895670+0000 mon.a (mon.0) 763 : cluster [DBG] mgrmap e24: y(active, since 5s), standbys: x 2024-09-17T09:27:19.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:18.978033+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.? 172.21.15.103:0/567347881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:27:19.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:18.978280+0000 mon.a (mon.0) 764 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T09:27:19.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:18.978398+0000 mon.a (mon.0) 765 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:27:19.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:18.979995+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.? 172.21.15.103:0/567347881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:27:19.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:18.980675+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.? 172.21.15.103:0/567347881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:27:19.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:18.980935+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.? 172.21.15.103:0/567347881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:27:20.718 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:19.424976+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:19.431094+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:19.455006+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:19.460146+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:19.899459+0000 mon.a (mon.0) 770 : cluster [DBG] mgrmap e25: y(active, since 6s), standbys: x 2024-09-17T09:27:20.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:19.982845+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:19.988530+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:19.993810+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:19.999001+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.719 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:19.424976+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.720 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:19.431094+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.720 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:19.455006+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.720 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:19.460146+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.720 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:19.899459+0000 mon.a (mon.0) 770 : cluster [DBG] mgrmap e25: y(active, since 6s), standbys: x 2024-09-17T09:27:20.720 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:19.982845+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.720 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:19.988530+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.720 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:19.993810+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.720 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:19.999001+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:19.424976+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:19.431094+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:19.455006+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:19.460146+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:19.899459+0000 mon.a (mon.0) 770 : cluster [DBG] mgrmap e25: y(active, since 6s), standbys: x 2024-09-17T09:27:20.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:19.982845+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:19.988530+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:19.993810+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:20.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:19.999001+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:21.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:19.879951+0000 mgr.y (mgr.25054) 10 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:21.761 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:19.879951+0000 mgr.y (mgr.25054) 10 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:21.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:19.879951+0000 mgr.y (mgr.25054) 10 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:23.344 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 4. 2024-09-17T09:27:23.345 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:23.345 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:27:23.345 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 podman[45943]: 2024-09-17 09:27:23.098947233 +0000 UTC m=+0.018596452 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:27:23.345 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 podman[45943]: 2024-09-17 09:27:23.237066308 +0000 UTC m=+0.156715507 container create c56b30d721d72e98dc4443111563989c8cf9a271d868d003ca14fa7951e12412 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:23.602 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:21.880632+0000 mgr.y (mgr.25054) 11 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T09:27:23.602 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 podman[45943]: 2024-09-17 09:27:23.495828607 +0000 UTC m=+0.415477810 container init c56b30d721d72e98dc4443111563989c8cf9a271d868d003ca14fa7951e12412 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:23.602 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 podman[45943]: 2024-09-17 09:27:23.498861523 +0000 UTC m=+0.418510720 container start c56b30d721d72e98dc4443111563989c8cf9a271d868d003ca14fa7951e12412 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:23.602 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.528Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:27:23.603 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.528Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:27:23.603 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.528Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:27:23.603 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.528Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:27:23.603 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.528Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:27:23.603 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.528Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:27:23.603 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.529Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:27:23.603 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.529Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:27:23.603 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.531Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:27:23.603 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.532Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2024-09-17T09:27:23.603 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.538Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2024-09-17T09:27:23.603 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.538Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2024-09-17T09:27:23.603 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.538Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2024-09-17T09:27:23.604 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.539Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2024-09-17T09:27:23.604 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.539Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2024-09-17T09:27:23.604 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.539Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2024-09-17T09:27:23.604 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.539Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=7.755358ms 2024-09-17T09:27:23.604 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.541Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:27:23.604 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.541Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:27:23.604 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.541Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:27:23.604 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.541Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:27:23.604 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.541Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:27:23.604 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.541Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:27:23.604 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.541Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:27:23.604 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.541Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:27:23.604 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.541Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:27:23.605 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.541Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:27:23.605 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.541Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:27:23.605 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.541Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:27:23.605 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=info ts=2024-09-17T09:27:23.541Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:27:23.605 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[45958]: level=error ts=2024-09-17T09:27:23.542Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:27:23.605 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 bash[45943]: c56b30d721d72e98dc4443111563989c8cf9a271d868d003ca14fa7951e12412 2024-09-17T09:27:23.605 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:23.605 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:23 smithi103 podman[45975]: 2024-09-17 09:27:23.601340804 +0000 UTC m=+0.050401229 container died c56b30d721d72e98dc4443111563989c8cf9a271d868d003ca14fa7951e12412 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:23.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:21.880632+0000 mgr.y (mgr.25054) 11 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T09:27:23.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:21.880632+0000 mgr.y (mgr.25054) 11 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T09:27:24.505 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:24 smithi103 podman[45975]: 2024-09-17 09:27:24.226937588 +0000 UTC m=+0.675998007 container remove c56b30d721d72e98dc4443111563989c8cf9a271d868d003ca14fa7951e12412 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:24.506 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:24 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:27:24.506 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:24 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:27:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:23.721127+0000 mon.a (mon.0) 775 : audit [DBG] from='client.? 172.21.15.29:0/1972270756' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:24.060602+0000 mon.c (mon.1) 58 : audit [INF] from='client.? 172.21.15.29:0/772211776' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: access denied 2024-09-17T09:27:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:24.233113+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:24.238729+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:24.240362+0000 mon.c (mon.1) 59 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:24.241026+0000 mon.c (mon.1) 60 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:27:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:24.246129+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:24.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:24.319616+0000 mon.c (mon.1) 61 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:27:24.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:24.319975+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:27:24.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:24.325098+0000 mon.c (mon.1) 62 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:24.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:23.721127+0000 mon.a (mon.0) 775 : audit [DBG] from='client.? 172.21.15.29:0/1972270756' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:24.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:24.060602+0000 mon.c (mon.1) 58 : audit [INF] from='client.? 172.21.15.29:0/772211776' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: access denied 2024-09-17T09:27:24.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:24.233113+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:24.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:24.238729+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:24.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:24.240362+0000 mon.c (mon.1) 59 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:24.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:24.241026+0000 mon.c (mon.1) 60 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:27:24.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:24.246129+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:24.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:24.319616+0000 mon.c (mon.1) 61 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:27:24.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:24.319975+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:27:24.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:24.325098+0000 mon.c (mon.1) 62 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:24.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:23.721127+0000 mon.a (mon.0) 775 : audit [DBG] from='client.? 172.21.15.29:0/1972270756' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:24.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:24.060602+0000 mon.c (mon.1) 58 : audit [INF] from='client.? 172.21.15.29:0/772211776' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: access denied 2024-09-17T09:27:24.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:24.233113+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:24.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:24.238729+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:24.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:24.240362+0000 mon.c (mon.1) 59 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:24.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:24.241026+0000 mon.c (mon.1) 60 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:27:24.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:24.246129+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:24.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:24.319616+0000 mon.c (mon.1) 61 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:27:24.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:24.319975+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:27:24.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:24 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:24.325098+0000 mon.c (mon.1) 62 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:25.661 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:23.880980+0000 mgr.y (mgr.25054) 12 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T09:27:25.661 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:24.241586+0000 mgr.y (mgr.25054) 13 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:27:25.661 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:24.242026+0000 mgr.y (mgr.25054) 14 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T09:27:25.661 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:24.319186+0000 mgr.y (mgr.25054) 15 : cephadm [INF] Reconfiguring iscsi.foo.smithi029.gellkh (dependencies changed)... 2024-09-17T09:27:25.661 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:24.325822+0000 mgr.y (mgr.25054) 16 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:27:25.662 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:24.463887+0000 mon.a (mon.0) 780 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:27:25.662 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:23.880980+0000 mgr.y (mgr.25054) 12 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T09:27:25.662 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:24.241586+0000 mgr.y (mgr.25054) 13 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:27:25.662 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:24.242026+0000 mgr.y (mgr.25054) 14 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T09:27:25.662 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:24.319186+0000 mgr.y (mgr.25054) 15 : cephadm [INF] Reconfiguring iscsi.foo.smithi029.gellkh (dependencies changed)... 2024-09-17T09:27:25.662 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:24.325822+0000 mgr.y (mgr.25054) 16 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:27:25.662 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:24.463887+0000 mon.a (mon.0) 780 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:27:25.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:23.880980+0000 mgr.y (mgr.25054) 12 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T09:27:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:24.241586+0000 mgr.y (mgr.25054) 13 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:27:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:24.242026+0000 mgr.y (mgr.25054) 14 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-17T09:27:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:24.319186+0000 mgr.y (mgr.25054) 15 : cephadm [INF] Reconfiguring iscsi.foo.smithi029.gellkh (dependencies changed)... 2024-09-17T09:27:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:24.325822+0000 mgr.y (mgr.25054) 16 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:27:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:24.463887+0000 mon.a (mon.0) 780 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:27:27.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:26.242742+0000 mgr.y (mgr.25054) 17 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:27:27.739 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:26.242742+0000 mgr.y (mgr.25054) 17 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:27:27.739 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:27 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:26.242742+0000 mgr.y (mgr.25054) 17 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:27:29.275 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:28.125518+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:29.276 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:28.131503+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:29.276 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:28.134134+0000 mgr.y (mgr.25054) 18 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:27:29.276 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:28.243132+0000 mgr.y (mgr.25054) 19 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:27:29.276 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:28.387918+0000 mgr.y (mgr.25054) 20 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:27:29.276 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:28.595659+0000 mon.a (mon.0) 783 : audit [DBG] from='client.? 172.21.15.29:0/1756338181' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:29.276 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:28.928112+0000 mon.a (mon.0) 784 : audit [INF] from='client.? 172.21.15.29:0/182740465' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: access denied 2024-09-17T09:27:29.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:28.125518+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:29.277 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:28.131503+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:29.277 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:28.134134+0000 mgr.y (mgr.25054) 18 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:27:29.277 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:28.243132+0000 mgr.y (mgr.25054) 19 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:27:29.277 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:28.387918+0000 mgr.y (mgr.25054) 20 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:27:29.277 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:28.595659+0000 mon.a (mon.0) 783 : audit [DBG] from='client.? 172.21.15.29:0/1756338181' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:29.277 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:29 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:28.928112+0000 mon.a (mon.0) 784 : audit [INF] from='client.? 172.21.15.29:0/182740465' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: access denied 2024-09-17T09:27:29.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:28.125518+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:29.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:28.131503+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:29.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:28.134134+0000 mgr.y (mgr.25054) 18 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:27:29.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:28.243132+0000 mgr.y (mgr.25054) 19 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:27:29.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:28.387918+0000 mgr.y (mgr.25054) 20 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:27:29.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:28.595659+0000 mon.a (mon.0) 783 : audit [DBG] from='client.? 172.21.15.29:0/1756338181' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:29.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:29 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:28.928112+0000 mon.a (mon.0) 784 : audit [INF] from='client.? 172.21.15.29:0/182740465' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: access denied 2024-09-17T09:27:31.555 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:31 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:30.244002+0000 mgr.y (mgr.25054) 21 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:27:31.556 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:31 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:31.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:30.244002+0000 mgr.y (mgr.25054) 21 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:27:31.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:30.244002+0000 mgr.y (mgr.25054) 21 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:27:31.917 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:31 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:27:31.917 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:31 smithi103 podman[46235]: 2024-09-17 09:27:31.650618219 +0000 UTC m=+0.017562364 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:27:31.918 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:31 smithi103 podman[46235]: 2024-09-17 09:27:31.789253762 +0000 UTC m=+0.156197960 container create 5c9497207611fa854ede7711cc6226b5e5d4d3620976160c6aae44b51096cd88 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:32.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 podman[46235]: 2024-09-17 09:27:32.015992572 +0000 UTC m=+0.382936708 container init 5c9497207611fa854ede7711cc6226b5e5d4d3620976160c6aae44b51096cd88 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:32.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 podman[46235]: 2024-09-17 09:27:32.018742548 +0000 UTC m=+0.385686681 container start 5c9497207611fa854ede7711cc6226b5e5d4d3620976160c6aae44b51096cd88 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:32.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.047Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:27:32.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.048Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:27:32.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.048Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:27:32.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.048Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:27:32.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.048Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:27:32.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.048Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:27:32.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.049Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:27:32.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.049Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:27:32.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.052Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:27:32.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.052Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=7 2024-09-17T09:27:32.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.058Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=7 2024-09-17T09:27:32.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.058Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=7 2024-09-17T09:27:32.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.058Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=7 2024-09-17T09:27:32.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.058Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=7 2024-09-17T09:27:32.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.059Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=7 2024-09-17T09:27:32.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.059Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=7 2024-09-17T09:27:32.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.060Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=7 2024-09-17T09:27:32.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.060Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=8.088132ms 2024-09-17T09:27:32.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.062Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:27:32.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.062Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:27:32.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.062Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:27:32.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.062Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:27:32.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.062Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:27:32.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.062Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:27:32.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.062Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:27:32.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.062Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:27:32.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.062Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:27:32.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.062Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:27:32.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.062Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:27:32.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.062Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:27:32.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=info ts=2024-09-17T09:27:32.062Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:27:32.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[46250]: level=error ts=2024-09-17T09:27:32.062Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:27:32.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 bash[46235]: 5c9497207611fa854ede7711cc6226b5e5d4d3620976160c6aae44b51096cd88 2024-09-17T09:27:32.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:32.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 podman[46267]: 2024-09-17 09:27:32.120511075 +0000 UTC m=+0.050866449 container died 5c9497207611fa854ede7711cc6226b5e5d4d3620976160c6aae44b51096cd88 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:32.729 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 podman[46267]: 2024-09-17 09:27:32.729330765 +0000 UTC m=+0.659686129 container remove 5c9497207611fa854ede7711cc6226b5e5d4d3620976160c6aae44b51096cd88 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:33.005 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:27:33.005 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:32 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:27:33.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:32.137853+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:33.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:32.144280+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:33.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:32.148629+0000 mon.c (mon.1) 63 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:27:33.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:32.149263+0000 mgr.y (mgr.25054) 22 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:27:33.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:32.152978+0000 mon.c (mon.1) 64 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:27:33.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:32.153391+0000 mgr.y (mgr.25054) 23 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:27:33.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:32.158572+0000 mon.c (mon.1) 65 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:27:33.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:32.159020+0000 mgr.y (mgr.25054) 24 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:27:33.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:32.198704+0000 mon.c (mon.1) 66 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:27:33.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:32.244379+0000 mgr.y (mgr.25054) 25 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2024-09-17T09:27:33.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:32.137853+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:33.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:32.144280+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:33.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:32.148629+0000 mon.c (mon.1) 63 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:27:33.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:32.149263+0000 mgr.y (mgr.25054) 22 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:27:33.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:32.152978+0000 mon.c (mon.1) 64 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:27:33.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:32.153391+0000 mgr.y (mgr.25054) 23 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:27:33.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:32.158572+0000 mon.c (mon.1) 65 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:27:33.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:32.159020+0000 mgr.y (mgr.25054) 24 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:27:33.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:32.198704+0000 mon.c (mon.1) 66 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:27:33.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:32.244379+0000 mgr.y (mgr.25054) 25 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2024-09-17T09:27:33.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:32.137853+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:33.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:32.144280+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:33.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:32.148629+0000 mon.c (mon.1) 63 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:27:33.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:32.149263+0000 mgr.y (mgr.25054) 22 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:27:33.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:32.152978+0000 mon.c (mon.1) 64 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:27:33.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:32.153391+0000 mgr.y (mgr.25054) 23 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:27:33.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:32.158572+0000 mon.c (mon.1) 65 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:27:33.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:32.159020+0000 mgr.y (mgr.25054) 24 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:27:33.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:32.198704+0000 mon.c (mon.1) 66 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:27:33.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:32.244379+0000 mgr.y (mgr.25054) 25 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2024-09-17T09:27:33.618 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:27:34.273 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:27:34.273 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (71s) 14s ago 5m 11.2M - 0.20.0 0881eb8f169f 7bd158a6d65c 2024-09-17T09:27:34.273 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (41s) 14s ago 5m 28.6M - 6.7.4 557c83e11646 e361816f7c14 2024-09-17T09:27:34.273 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 starting - - - - 2024-09-17T09:27:34.273 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443 running (107s) 14s ago 9m 96.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 d35594bdef28 2024-09-17T09:27:34.273 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (34s) 14s ago 11m 454M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:27:34.274 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (11m) 14s ago 11m 53.9M 2048M 16.2.0 24ecd6d5f14c f3476652dfdd 2024-09-17T09:27:34.274 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (9m) 14s ago 9m 46.0M 2048M 16.2.0 24ecd6d5f14c 203d1f26d7b8 2024-09-17T09:27:34.274 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (10m) 14s ago 10m 45.2M 2048M 16.2.0 24ecd6d5f14c aa05f3e5656a 2024-09-17T09:27:34.274 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (80s) 14s ago 6m 12.9M - 0.18.1 e5a616e4b9cf 34bfa263169f 2024-09-17T09:27:34.274 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (53s) 14s ago 6m 2558k - 0.18.1 e5a616e4b9cf 1adf1b5baad1 2024-09-17T09:27:34.274 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (9m) 14s ago 9m 47.4M 4096M 16.2.0 24ecd6d5f14c 991731ce08b5 2024-09-17T09:27:34.274 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (9m) 14s ago 9m 45.6M 4096M 16.2.0 24ecd6d5f14c d5bce63fadc8 2024-09-17T09:27:34.274 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (8m) 14s ago 8m 42.6M 4096M 16.2.0 24ecd6d5f14c e9678d2f1685 2024-09-17T09:27:34.274 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (8m) 14s ago 8m 49.1M 4096M 16.2.0 24ecd6d5f14c dcf7122a7d99 2024-09-17T09:27:34.274 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (8m) 14s ago 8m 48.6M 4096M 16.2.0 24ecd6d5f14c 048c13025d32 2024-09-17T09:27:34.274 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (7m) 14s ago 7m 47.2M 4096M 16.2.0 24ecd6d5f14c 10a64919fbc9 2024-09-17T09:27:34.274 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (7m) 14s ago 7m 45.7M 4096M 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:27:34.274 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (7m) 14s ago 7m 51.8M 4096M 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:27:34.275 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 starting - - - - 2024-09-17T09:27:34.275 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (5m) 14s ago 5m 39.7M - 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:27:34.275 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (5m) 14s ago 5m 39.9M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:27:34.275 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (5m) 14s ago 4m 37.4M - 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:27:34.275 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (5m) 14s ago 5m 36.7M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:27:34.499 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:33.926242+0000 mon.c (mon.1) 67 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:27:34.499 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:33.926713+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:27:34.499 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:33.927708+0000 mon.c (mon.1) 68 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:27:34.499 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:34 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:33.928440+0000 mon.c (mon.1) 69 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:34.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:33.926242+0000 mon.c (mon.1) 67 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:27:34.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:33.926713+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:27:34.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:33.927708+0000 mon.c (mon.1) 68 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:27:34.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:33.928440+0000 mon.c (mon.1) 69 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:34.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:33.926242+0000 mon.c (mon.1) 67 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:27:34.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:33.926713+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:27:34.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:33.927708+0000 mon.c (mon.1) 68 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:27:34.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:33.928440+0000 mon.c (mon.1) 69 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:34.671 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:27:34.671 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:27:34.671 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3 2024-09-17T09:27:34.671 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:27:34.671 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:27:34.671 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:27:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:27:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:27:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T09:27:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:27:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": {}, 2024-09-17T09:27:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:27:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:27:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:27:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:27:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 15, 2024-09-17T09:27:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:27:34.672 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:27:34.672 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:27:35.006 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:27:35.006 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:27:35.006 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:27:35.006 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:27:35.006 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [ 2024-09-17T09:27:35.006 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr" 2024-09-17T09:27:35.006 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:27:35.007 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "2/23 daemons upgraded", 2024-09-17T09:27:35.007 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Currently upgrading mgr daemons", 2024-09-17T09:27:35.007 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:27:35.007 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:27:35.407 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:27:35.408 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:27:35.408 INFO:teuthology.orchestra.run.smithi029.stdout: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:27:35.408 INFO:teuthology.orchestra.run.smithi029.stdout: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:27:35.547 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:33.604219+0000 mgr.y (mgr.25054) 26 : audit [DBG] from='client.25198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:35.547 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:33.925826+0000 mgr.y (mgr.25054) 27 : cephadm [INF] Upgrade: Updating mgr.x 2024-09-17T09:27:35.547 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:33.929158+0000 mgr.y (mgr.25054) 28 : cephadm [INF] Deploying daemon mgr.x on smithi103 2024-09-17T09:27:35.547 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:33.937080+0000 mgr.y (mgr.25054) 29 : audit [DBG] from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:35.547 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:34.244742+0000 mgr.y (mgr.25054) 30 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2024-09-17T09:27:35.548 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:34.264108+0000 mgr.y (mgr.25054) 31 : audit [DBG] from='client.25210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:35.548 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:34.670660+0000 mon.c (mon.1) 70 : audit [DBG] from='client.? 172.21.15.29:0/1521621276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:27:35.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:33.604219+0000 mgr.y (mgr.25054) 26 : audit [DBG] from='client.25198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:35.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:33.925826+0000 mgr.y (mgr.25054) 27 : cephadm [INF] Upgrade: Updating mgr.x 2024-09-17T09:27:35.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:33.929158+0000 mgr.y (mgr.25054) 28 : cephadm [INF] Deploying daemon mgr.x on smithi103 2024-09-17T09:27:35.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:33.937080+0000 mgr.y (mgr.25054) 29 : audit [DBG] from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:35.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:34.244742+0000 mgr.y (mgr.25054) 30 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2024-09-17T09:27:35.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:34.264108+0000 mgr.y (mgr.25054) 31 : audit [DBG] from='client.25210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:35.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:34.670660+0000 mon.c (mon.1) 70 : audit [DBG] from='client.? 172.21.15.29:0/1521621276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:27:35.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:33.604219+0000 mgr.y (mgr.25054) 26 : audit [DBG] from='client.25198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:35.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:33.925826+0000 mgr.y (mgr.25054) 27 : cephadm [INF] Upgrade: Updating mgr.x 2024-09-17T09:27:35.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:33.929158+0000 mgr.y (mgr.25054) 28 : cephadm [INF] Deploying daemon mgr.x on smithi103 2024-09-17T09:27:35.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:33.937080+0000 mgr.y (mgr.25054) 29 : audit [DBG] from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:35.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:34.244742+0000 mgr.y (mgr.25054) 30 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 197 B/s rd, 0 op/s 2024-09-17T09:27:35.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:34.264108+0000 mgr.y (mgr.25054) 31 : audit [DBG] from='client.25210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:35.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:34.670660+0000 mon.c (mon.1) 70 : audit [DBG] from='client.? 172.21.15.29:0/1521621276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:27:36.005 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:35 smithi103 systemd[1]: Stopping Ceph mgr.x for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:27:36.005 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:35 smithi103 podman[46576]: 2024-09-17 09:27:35.771621185 +0000 UTC m=+0.139089182 container died d35594bdef289e64437c1e6b911c9ce4c90ee72bb7ea0bbea38a117e83122c94 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:27:36.292 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:36 smithi103 podman[46576]: 2024-09-17 09:27:36.107135503 +0000 UTC m=+0.474603492 container cleanup d35594bdef289e64437c1e6b911c9ce4c90ee72bb7ea0bbea38a117e83122c94 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-17T09:27:36.292 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:36 smithi103 bash[46576]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x 2024-09-17T09:27:36.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:35.005458+0000 mgr.y (mgr.25054) 32 : audit [DBG] from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:36.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:35.407424+0000 mon.c (mon.1) 71 : audit [DBG] from='client.? 172.21.15.29:0/673236909' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:27:36.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:35.005458+0000 mgr.y (mgr.25054) 32 : audit [DBG] from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:36.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:35.407424+0000 mon.c (mon.1) 71 : audit [DBG] from='client.? 172.21.15.29:0/673236909' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:27:36.674 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:35.005458+0000 mgr.y (mgr.25054) 32 : audit [DBG] from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:27:36.674 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:36 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:35.407424+0000 mon.c (mon.1) 71 : audit [DBG] from='client.? 172.21.15.29:0/673236909' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:27:36.674 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:36 smithi103 podman[46589]: 2024-09-17 09:27:36.317036729 +0000 UTC m=+0.543334458 container remove d35594bdef289e64437c1e6b911c9ce4c90ee72bb7ea0bbea38a117e83122c94 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr.x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-17T09:27:36.674 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:36 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:27:37.005 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:36 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-17T09:27:37.006 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:36 smithi103 systemd[1]: Stopped Ceph mgr.x for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:37.006 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:36 smithi103 systemd[1]: Starting Ceph mgr.x for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:27:37.374 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:36.245505+0000 mgr.y (mgr.25054) 33 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T09:27:37.374 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:37 smithi103 podman[46729]: 2024-09-17 09:27:37.00259134 +0000 UTC m=+0.016163062 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:27:37.374 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:37 smithi103 podman[46729]: 2024-09-17 09:27:37.127046588 +0000 UTC m=+0.140618296 container create 88340aeba3bad77e143f84fae732ba60582d6bd18d88695c4cb92145f9b2370a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-17T09:27:37.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:36.245505+0000 mgr.y (mgr.25054) 33 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T09:27:37.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:37 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:36.245505+0000 mgr.y (mgr.25054) 33 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-17T09:27:37.712 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:37 smithi103 podman[46729]: 2024-09-17 09:27:37.373478992 +0000 UTC m=+0.387050706 container init 88340aeba3bad77e143f84fae732ba60582d6bd18d88695c4cb92145f9b2370a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-17T09:27:37.712 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:37 smithi103 podman[46729]: 2024-09-17 09:27:37.376536334 +0000 UTC m=+0.390108040 container start 88340aeba3bad77e143f84fae732ba60582d6bd18d88695c4cb92145f9b2370a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:27:37.712 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:37 smithi103 ceph-mgr[46749]: -- 172.21.15.103:0/4094018491 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ff04f44d00 con 0x55ff04f18800 2024-09-17T09:27:37.712 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:37 smithi103 bash[46729]: 88340aeba3bad77e143f84fae732ba60582d6bd18d88695c4cb92145f9b2370a 2024-09-17T09:27:37.712 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:37 smithi103 systemd[1]: Started Ceph mgr.x for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:37.712 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:37.492+0000 7fa257a95040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:27:37.712 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:37.552+0000 7fa257a95040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:27:37.712 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:37.710+0000 7fa257a95040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:27:38.005 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:37.782+0000 7fa257a95040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:27:38.005 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:37 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:37.936+0000 7fa257a95040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:27:38.448 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:38.119+0000 7fa257a95040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:27:38.449 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:38.446+0000 7fa257a95040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:27:38.704 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:38.511+0000 7fa257a95040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:27:38.704 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:38.701+0000 7fa257a95040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:27:38.704 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:37.545175+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:38.705 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:37.551512+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:38.705 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:37.793349+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:38.705 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:37.799007+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:38.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:37.545175+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:38.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:37.551512+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:38.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:37.793349+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:38.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:37.799007+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:38.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:37.545175+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:38.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:37.551512+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:38.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:37.793349+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:38.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:37.799007+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:39.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:38.245862+0000 mgr.y (mgr.25054) 34 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:39.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:38.245862+0000 mgr.y (mgr.25054) 34 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:40.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:38.245862+0000 mgr.y (mgr.25054) 34 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:40.006 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:39.638+0000 7fa257a95040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:27:40.006 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:39 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:39.703+0000 7fa257a95040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:27:40.884 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:40.581+0000 7fa257a95040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:27:40.885 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:40.718+0000 7fa257a95040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:27:40.885 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:40.883+0000 7fa257a95040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:27:41.168 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:40 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:40.946+0000 7fa257a95040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:27:41.168 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:41.098+0000 7fa257a95040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:27:41.169 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:41.167+0000 7fa257a95040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:27:41.453 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:41.229+0000 7fa257a95040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:27:41.453 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:41.387+0000 7fa257a95040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:27:41.453 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:41.452+0000 7fa257a95040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:27:41.747 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:40.246480+0000 mgr.y (mgr.25054) 35 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:41.748 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:41.520+0000 7fa257a95040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:27:41.748 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:41.617+0000 7fa257a95040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:27:41.748 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:41.680+0000 7fa257a95040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:27:41.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:40.246480+0000 mgr.y (mgr.25054) 35 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:41.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:40.246480+0000 mgr.y (mgr.25054) 35 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:42.005 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:41.746+0000 7fa257a95040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:27:42.929 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:42.534131+0000 mon.a (mon.0) 792 : audit [DBG] from='client.? 172.21.15.29:0/2528044311' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:42.929 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:42.534131+0000 mon.a (mon.0) 792 : audit [DBG] from='client.? 172.21.15.29:0/2528044311' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:42.945 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:42 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:42.534131+0000 mon.a (mon.0) 792 : audit [DBG] from='client.? 172.21.15.29:0/2528044311' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:42.946 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:42 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:42.558+0000 7fa257a95040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:27:42.946 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:42 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 1. 2024-09-17T09:27:42.946 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:42 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:42.946 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:42 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:27:43.201 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:42 smithi103 podman[47592]: 2024-09-17 09:27:42.848838685 +0000 UTC m=+0.017529498 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:27:43.201 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:42 smithi103 podman[47592]: 2024-09-17 09:27:42.973546305 +0000 UTC m=+0.142237107 container create d64fc399cc93a0501f5958498a1d8db622dd52db23b02f9f5f08ca64a4f59c47 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:43.201 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 podman[47592]: 2024-09-17 09:27:43.200599215 +0000 UTC m=+0.369290027 container init d64fc399cc93a0501f5958498a1d8db622dd52db23b02f9f5f08ca64a4f59c47 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:43.490 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 podman[47592]: 2024-09-17 09:27:43.204390445 +0000 UTC m=+0.373081248 container start d64fc399cc93a0501f5958498a1d8db622dd52db23b02f9f5f08ca64a4f59c47 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:43.491 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.234Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:27:43.491 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.234Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:27:43.491 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.235Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:27:43.491 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.235Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:27:43.491 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.235Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:27:43.491 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.235Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:27:43.491 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.236Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:27:43.491 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.236Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:27:43.491 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.238Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:27:43.491 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.238Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=8 2024-09-17T09:27:43.492 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.246Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=8 2024-09-17T09:27:43.492 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.246Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=8 2024-09-17T09:27:43.492 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.246Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=8 2024-09-17T09:27:43.492 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.246Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=8 2024-09-17T09:27:43.492 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.247Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=8 2024-09-17T09:27:43.492 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.247Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=8 2024-09-17T09:27:43.492 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.247Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=8 2024-09-17T09:27:43.492 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.247Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=8 2024-09-17T09:27:43.492 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.247Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=9.081513ms 2024-09-17T09:27:43.492 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.249Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:27:43.492 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.249Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:27:43.492 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.249Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:27:43.492 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.250Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:27:43.493 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.250Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:27:43.493 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.250Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:27:43.493 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.250Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:27:43.493 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.250Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:27:43.493 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.250Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:27:43.493 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.250Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:27:43.493 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.250Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:27:43.493 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.250Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:27:43.493 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=info ts=2024-09-17T09:27:43.250Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:27:43.493 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[47649]: level=error ts=2024-09-17T09:27:43.250Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:27:43.493 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 bash[47592]: d64fc399cc93a0501f5958498a1d8db622dd52db23b02f9f5f08ca64a4f59c47 2024-09-17T09:27:43.493 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:43.493 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 podman[47702]: 2024-09-17 09:27:43.298914902 +0000 UTC m=+0.040845192 container died d64fc399cc93a0501f5958498a1d8db622dd52db23b02f9f5f08ca64a4f59c47 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:43.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:42.246842+0000 mgr.y (mgr.25054) 36 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:43.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:42.560775+0000 mon.a (mon.0) 793 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T09:27:43.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:42.560901+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:27:43.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:42.561718+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.? 172.21.15.103:0/2000146433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:27:43.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:42.562134+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.? 172.21.15.103:0/2000146433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:27:43.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:42.562882+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.? 172.21.15.103:0/2000146433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:27:43.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:42.563141+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.? 172.21.15.103:0/2000146433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:27:43.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:42.759628+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:43.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:42.780287+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:43.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:42.924366+0000 mon.a (mon.0) 797 : audit [INF] from='client.? 172.21.15.29:0/3630906061' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: access denied 2024-09-17T09:27:43.716 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:42.246842+0000 mgr.y (mgr.25054) 36 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:43.716 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:42.560775+0000 mon.a (mon.0) 793 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T09:27:43.716 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:42.560901+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:27:43.716 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:42.561718+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.? 172.21.15.103:0/2000146433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:27:43.716 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:42.562134+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.? 172.21.15.103:0/2000146433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:27:43.716 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:42.562882+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.? 172.21.15.103:0/2000146433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:27:43.716 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:42.563141+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.? 172.21.15.103:0/2000146433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:27:43.716 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:42.759628+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:43.716 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:42.780287+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:43.716 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:42.924366+0000 mon.a (mon.0) 797 : audit [INF] from='client.? 172.21.15.29:0/3630906061' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: access denied 2024-09-17T09:27:43.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:42.246842+0000 mgr.y (mgr.25054) 36 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:43.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:42.560775+0000 mon.a (mon.0) 793 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T09:27:43.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:42.560901+0000 mon.a (mon.0) 794 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:27:43.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:42.561718+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.? 172.21.15.103:0/2000146433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:27:43.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:42.562134+0000 mon.b (mon.2) 83 : audit [DBG] from='mgr.? 172.21.15.103:0/2000146433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:27:43.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:42.562882+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.? 172.21.15.103:0/2000146433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:27:43.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:42.563141+0000 mon.b (mon.2) 85 : audit [DBG] from='mgr.? 172.21.15.103:0/2000146433' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:27:43.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:42.759628+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:43.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:42.780287+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:43.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:43 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:42.924366+0000 mon.a (mon.0) 797 : audit [INF] from='client.? 172.21.15.29:0/3630906061' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: access denied 2024-09-17T09:27:44.255 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 podman[47702]: 2024-09-17 09:27:43.976908434 +0000 UTC m=+0.718838715 container remove d64fc399cc93a0501f5958498a1d8db622dd52db23b02f9f5f08ca64a4f59c47 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:44.255 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:27:44.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:43 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:27:44.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:43.565050+0000 mon.a (mon.0) 798 : cluster [DBG] mgrmap e26: y(active, since 30s), standbys: x 2024-09-17T09:27:44.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:44.089711+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:44.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:44.095849+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:45.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:43.565050+0000 mon.a (mon.0) 798 : cluster [DBG] mgrmap e26: y(active, since 30s), standbys: x 2024-09-17T09:27:45.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:44.089711+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:45.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:44.095849+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:45.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:43.565050+0000 mon.a (mon.0) 798 : cluster [DBG] mgrmap e26: y(active, since 30s), standbys: x 2024-09-17T09:27:45.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:44.089711+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:45.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:44 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:44.095849+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:45.682 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:44.247259+0000 mgr.y (mgr.25054) 37 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:45.682 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:44.987294+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:45.682 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:44.992572+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:46.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:44.247259+0000 mgr.y (mgr.25054) 37 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:46.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:44.987294+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:46.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:44.992572+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:46.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:44.247259+0000 mgr.y (mgr.25054) 37 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:27:46.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:44.987294+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:46.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:45 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:44.992572+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:47.826 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:46.248139+0000 mgr.y (mgr.25054) 38 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:47.826 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.342769+0000 mon.a (mon.0) 803 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:47.826 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.348927+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:47.826 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.350417+0000 mon.c (mon.1) 72 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:47.826 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.351333+0000 mon.c (mon.1) 73 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:27:47.826 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.357684+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:47.826 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.477607+0000 mon.c (mon.1) 74 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:27:47.827 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.480353+0000 mon.c (mon.1) 75 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:27:47.827 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.485490+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:47.827 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.486513+0000 mon.c (mon.1) 76 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T09:27:47.827 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.486793+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T09:27:47.827 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.490208+0000 mon.a (mon.0) 808 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-17T09:27:47.827 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.490956+0000 mon.c (mon.1) 77 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T09:27:47.827 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.491210+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T09:27:47.827 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.494424+0000 mon.a (mon.0) 810 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-17T09:27:47.827 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.495591+0000 mon.c (mon.1) 78 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T09:27:47.827 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:47.496271+0000 mon.c (mon.1) 79 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-17T09:27:47.827 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:46.248139+0000 mgr.y (mgr.25054) 38 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:47.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.342769+0000 mon.a (mon.0) 803 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:47.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.348927+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:47.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.350417+0000 mon.c (mon.1) 72 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:47.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.351333+0000 mon.c (mon.1) 73 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:27:47.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.357684+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:47.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.477607+0000 mon.c (mon.1) 74 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:27:47.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.480353+0000 mon.c (mon.1) 75 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:27:47.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.485490+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:47.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.486513+0000 mon.c (mon.1) 76 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T09:27:47.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.486793+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T09:27:47.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.490208+0000 mon.a (mon.0) 808 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-17T09:27:47.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.490956+0000 mon.c (mon.1) 77 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T09:27:47.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.491210+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T09:27:47.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.494424+0000 mon.a (mon.0) 810 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-17T09:27:47.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.495591+0000 mon.c (mon.1) 78 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T09:27:47.829 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:47.496271+0000 mon.c (mon.1) 79 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-17T09:27:48.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:46.248139+0000 mgr.y (mgr.25054) 38 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:27:48.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.342769+0000 mon.a (mon.0) 803 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:48.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.348927+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:48.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.350417+0000 mon.c (mon.1) 72 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:48.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.351333+0000 mon.c (mon.1) 73 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:27:48.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.357684+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:48.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.477607+0000 mon.c (mon.1) 74 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:27:48.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.480353+0000 mon.c (mon.1) 75 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:27:48.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.485490+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:48.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.486513+0000 mon.c (mon.1) 76 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T09:27:48.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.486793+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T09:27:48.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.490208+0000 mon.a (mon.0) 808 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-17T09:27:48.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.490956+0000 mon.c (mon.1) 77 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T09:27:48.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.491210+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T09:27:48.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.494424+0000 mon.a (mon.0) 810 : audit [INF] from='mgr.25054 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-17T09:27:48.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.495591+0000 mon.c (mon.1) 78 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T09:27:48.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:47.496271+0000 mon.c (mon.1) 79 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-17T09:27:48.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:47.352111+0000 mgr.y (mgr.25054) 39 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:27:48.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:47.352700+0000 mgr.y (mgr.25054) 40 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 92 B/s rd, 0 op/s 2024-09-17T09:27:48.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:47.352902+0000 mgr.y (mgr.25054) 41 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-17T09:27:48.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:47.481027+0000 mgr.y (mgr.25054) 42 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-09-17T09:27:48.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:47.496764+0000 mgr.y (mgr.25054) 43 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2024-09-17T09:27:48.862 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:47.352111+0000 mgr.y (mgr.25054) 39 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:27:48.862 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:47.352700+0000 mgr.y (mgr.25054) 40 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 92 B/s rd, 0 op/s 2024-09-17T09:27:48.862 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:47.352902+0000 mgr.y (mgr.25054) 41 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-17T09:27:48.862 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:47.481027+0000 mgr.y (mgr.25054) 42 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-09-17T09:27:48.863 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:47.496764+0000 mgr.y (mgr.25054) 43 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2024-09-17T09:27:48.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:47.352111+0000 mgr.y (mgr.25054) 39 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:27:48.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:47.352700+0000 mgr.y (mgr.25054) 40 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 92 B/s rd, 0 op/s 2024-09-17T09:27:48.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:47.352902+0000 mgr.y (mgr.25054) 41 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-17T09:27:48.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:47.481027+0000 mgr.y (mgr.25054) 42 : cephadm [INF] Upgrade: Setting container_image for all mgr 2024-09-17T09:27:48.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:48 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:47.496764+0000 mgr.y (mgr.25054) 43 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2024-09-17T09:27:50.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:48.934002+0000 mgr.y (mgr.25054) 44 : cephadm [INF] Upgrade: Updating mon.a 2024-09-17T09:27:50.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:48.939225+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:50.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:48.940231+0000 mon.c (mon.1) 80 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:27:50.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:48.940883+0000 mon.c (mon.1) 81 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:27:50.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:48.941567+0000 mon.c (mon.1) 82 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:50.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:27:48.942392+0000 mgr.y (mgr.25054) 45 : cephadm [INF] Deploying daemon mon.a on smithi029 2024-09-17T09:27:50.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:49.353430+0000 mgr.y (mgr.25054) 46 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-17T09:27:50.266 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:48.934002+0000 mgr.y (mgr.25054) 44 : cephadm [INF] Upgrade: Updating mon.a 2024-09-17T09:27:50.266 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:48.939225+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:50.266 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:48.940231+0000 mon.c (mon.1) 80 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:27:50.266 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:48.940883+0000 mon.c (mon.1) 81 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:27:50.266 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:48.941567+0000 mon.c (mon.1) 82 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:50.266 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:27:48.942392+0000 mgr.y (mgr.25054) 45 : cephadm [INF] Deploying daemon mon.a on smithi029 2024-09-17T09:27:50.266 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:49.353430+0000 mgr.y (mgr.25054) 46 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-17T09:27:50.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:48.934002+0000 mgr.y (mgr.25054) 44 : cephadm [INF] Upgrade: Updating mon.a 2024-09-17T09:27:50.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:48.939225+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.25054 ' entity='mgr.y' 2024-09-17T09:27:50.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:48.940231+0000 mon.c (mon.1) 80 : audit [INF] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:27:50.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:48.940883+0000 mon.c (mon.1) 81 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:27:50.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: audit 2024-09-17T09:27:48.941567+0000 mon.c (mon.1) 82 : audit [DBG] from='mgr.25054 172.21.15.29:0/355855631' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:27:50.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cephadm 2024-09-17T09:27:48.942392+0000 mgr.y (mgr.25054) 45 : cephadm [INF] Deploying daemon mon.a on smithi029 2024-09-17T09:27:50.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:49 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: cluster 2024-09-17T09:27:49.353430+0000 mgr.y (mgr.25054) 46 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-17T09:27:50.975 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:50 smithi029 systemd[1]: Stopping Ceph mon.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:27:50.975 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:27:50.722+0000 7fcd94b40700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-17T09:27:50.975 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:50 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a[23620]: debug 2024-09-17T09:27:50.722+0000 7fcd94b40700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-17T09:27:51.268 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:50 smithi029 podman[58150]: 2024-09-17 09:27:50.978875959 +0000 UTC m=+0.364168332 container died f3476652dfdd603e0da45438ea41439f46b027ba5ee63edb952263d4c0be850f (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.license=GPLv2) 2024-09-17T09:27:51.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:51 smithi029 podman[58150]: 2024-09-17 09:27:51.366505189 +0000 UTC m=+0.751797552 container cleanup f3476652dfdd603e0da45438ea41439f46b027ba5ee63edb952263d4c0be850f (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=-16.2.0, ceph=True) 2024-09-17T09:27:51.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:51 smithi029 bash[58150]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a 2024-09-17T09:27:51.954 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:51 smithi029 podman[58163]: 2024-09-17 09:27:51.669736192 +0000 UTC m=+0.686426295 container remove f3476652dfdd603e0da45438ea41439f46b027ba5ee63edb952263d4c0be850f (image=quay.io/ceph/ceph:v16.2.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.a, RELEASE=HEAD, maintainer=Dimitri Savineau , GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-17T09:27:52.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:51 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-17T09:27:52.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:51 smithi029 systemd[1]: Stopped Ceph mon.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:52.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 systemd[1]: Starting Ceph mon.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:27:52.665 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 podman[58306]: 2024-09-17 09:27:52.300188239 +0000 UTC m=+0.016272987 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:27:52.665 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 podman[58306]: 2024-09-17 09:27:52.406901661 +0000 UTC m=+0.122986396 container create 88689aaf0f3f98c46cf717a6e8b89505e12255ebdcc0461e2dd3f3210ede43a4 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-17T09:27:52.665 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 podman[58306]: 2024-09-17 09:27:52.664484894 +0000 UTC m=+0.380569638 container init 88689aaf0f3f98c46cf717a6e8b89505e12255ebdcc0461e2dd3f3210ede43a4 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-09-17T09:27:53.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 podman[58306]: 2024-09-17 09:27:52.667975239 +0000 UTC m=+0.384059979 container start 88689aaf0f3f98c46cf717a6e8b89505e12255ebdcc0461e2dd3f3210ede43a4 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:27:53.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T09:27:53.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process ceph-mon, pid 2 2024-09-17T09:27:53.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: pidfile_write: ignore empty --pid-file 2024-09-17T09:27:53.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: load: jerasure load: lrc 2024-09-17T09:27:53.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: RocksDB version: 7.9.2 2024-09-17T09:27:53.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Git sha 0 2024-09-17T09:27:53.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Compile date 2024-09-16 15:15:13 2024-09-17T09:27:53.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: DB SUMMARY 2024-09-17T09:27:53.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: DB Session ID: ANHMUU2APWOEZ4VIUMOL 2024-09-17T09:27:53.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: CURRENT file: CURRENT 2024-09-17T09:27:53.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T09:27:53.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: MANIFEST file: MANIFEST-000008 size: 972 Bytes 2024-09-17T09:27:53.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000023.sst 2024-09-17T09:27:53.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000021.log size: 894168 ; 2024-09-17T09:27:53.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.error_if_exists: 0 2024-09-17T09:27:53.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.create_if_missing: 0 2024-09-17T09:27:53.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.paranoid_checks: 1 2024-09-17T09:27:53.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.env: 0x564425ceabc0 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.info_log: 0x564427c5ebe0 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.statistics: (nil) 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.use_fsync: 0 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_log_file_size: 0 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.allow_fallocate: 1 2024-09-17T09:27:53.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T09:27:53.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T09:27:53.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.use_direct_reads: 0 2024-09-17T09:27:53.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T09:27:53.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T09:27:53.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.db_log_dir: 2024-09-17T09:27:53.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.wal_dir: 2024-09-17T09:27:53.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T09:27:53.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T09:27:53.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T09:27:53.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.write_buffer_manager: 0x564427c6cb40 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T09:27:53.053 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.unordered_write: 0 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.row_cache: None 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.wal_filter: None 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.two_write_queues: 0 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.wal_compression: 0 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.atomic_flush: 0 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T09:27:53.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.log_readahead_size: 0 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_background_jobs: 2 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_background_compactions: -1 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_subcompactions: 1 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T09:27:53.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T09:27:53.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T09:27:53.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T09:27:53.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T09:27:53.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T09:27:53.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T09:27:53.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T09:27:53.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_open_files: -1 2024-09-17T09:27:53.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T09:27:53.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T09:27:53.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T09:27:53.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T09:27:53.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_background_flushes: -1 2024-09-17T09:27:53.056 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Compression algorithms supported: 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: kZSTD supported: 0 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: kXpressCompression supported: 0 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: kBZip2Compression supported: 0 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: kLZ4Compression supported: 1 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: kZlibCompression supported: 1 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: kSnappyCompression supported: 1 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T09:27:53.057 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.merge_operator: 2024-09-17T09:27:53.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compaction_filter: None 2024-09-17T09:27:53.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T09:27:53.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T09:27:53.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T09:27:53.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T09:27:53.058 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564427c5e7e0) 2024-09-17T09:27:53.058 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-09-17T09:27:53.058 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T09:27:53.058 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T09:27:53.058 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-09-17T09:27:53.058 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_type: 0 2024-09-17T09:27:53.058 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_index_type: 0 2024-09-17T09:27:53.058 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_shortening: 1 2024-09-17T09:27:53.059 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T09:27:53.059 INFO:journalctl@ceph.mon.a.smithi029.stdout: checksum: 4 2024-09-17T09:27:53.059 INFO:journalctl@ceph.mon.a.smithi029.stdout: no_block_cache: 0 2024-09-17T09:27:53.059 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache: 0x564427c53350 2024-09-17T09:27:53.059 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-09-17T09:27:53.059 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_options: 2024-09-17T09:27:53.059 INFO:journalctl@ceph.mon.a.smithi029.stdout: capacity : 536870912 2024-09-17T09:27:53.059 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_shard_bits : 4 2024-09-17T09:27:53.059 INFO:journalctl@ceph.mon.a.smithi029.stdout: strict_capacity_limit : 0 2024-09-17T09:27:53.059 INFO:journalctl@ceph.mon.a.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-09-17T09:27:53.059 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_compressed: (nil) 2024-09-17T09:27:53.059 INFO:journalctl@ceph.mon.a.smithi029.stdout: persistent_cache: (nil) 2024-09-17T09:27:53.059 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size: 4096 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size_deviation: 10 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_restart_interval: 16 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_block_restart_interval: 1 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: metadata_block_size: 4096 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: partition_filters: 0 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: use_delta_encoding: 1 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: filter_policy: bloomfilter 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: whole_key_filtering: 1 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: verify_compression: 0 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: format_version: 5 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_index_compression: 1 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_align: 0 2024-09-17T09:27:53.060 INFO:journalctl@ceph.mon.a.smithi029.stdout: max_auto_readahead_size: 262144 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout: prepopulate_block_cache: 0 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout: initial_auto_readahead_size: 8192 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compression: NoCompression 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.num_levels: 7 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T09:27:53.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T09:27:53.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T09:27:53.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T09:27:53.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T09:27:53.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T09:27:53.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T09:27:53.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T09:27:53.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T09:27:53.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T09:27:53.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T09:27:53.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T09:27:53.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T09:27:53.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T09:27:53.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T09:27:53.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T09:27:53.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.inplace_update_support: 0 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.bloom_locality: 0 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.max_successive_merges: 0 2024-09-17T09:27:53.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.ttl: 2592000 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.enable_blob_files: false 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.min_blob_size: 0 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T09:27:53.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 23.sst 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000008 succeeded,manifest_file_number is 8, next_file_number is 25, last_sequence is 11352, log_number is 21,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 21 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: be78e775-e828-490b-8f4b-c45fc9153a70 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726565272695977, "job": 1, "event": "recovery_started", "wal_files": [21]} 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #21 mode 2 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726565272699823, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 26, "file_size": 868678, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11346, "largest_seqno": 11546, "table_properties": {"data_size": 866509, "index_size": 875, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 261, "raw_key_size": 2289, "raw_average_key_size": 25, "raw_value_size": 863989, "raw_average_value_size": 9599, "num_data_blocks": 38, "num_entries": 90, "num_filter_entries": 90, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726565272, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "be78e775-e828-490b-8f4b-c45fc9153a70", "db_session_id": "ANHMUU2APWOEZ4VIUMOL", "orig_file_number": 26, "seqno_to_time_mapping": "N/A"}} 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726565272699932, "job": 1, "event": "recovery_finished"} 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [db/version_set.cc:5047] Creating manifest 28 2024-09-17T09:27:53.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000021.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564427c7ee00 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: DB pointer 0x564427d72000 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** DB Stats ** 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:27:53.068 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: L0 1/0 848.32 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 326.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: L6 1/0 24.60 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: Sum 2/0 25.43 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 326.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 326.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 326.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-09-17T09:27:53.069 INFO:journalctl@ceph.mon.a.smithi029.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative compaction: 0.00 GB write, 132.04 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval compaction: 0.00 GB write, 132.04 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache BinnedLRUCache@0x564427c53350#2 capacity: 512.00 MB usage: 1.27 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.31 KB,5.96046e-05%) IndexBlock(1,0.95 KB,0.000181794%) Misc(1,0.00 KB,0%) 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-09-17T09:27:53.070 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: starting mon.a rank 0 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: mon.a@-1(???) e3 preinit fsid 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: mon.a@-1(???).mds e1 new map 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: mon.a@-1(???).mds e1 print_map 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout: e1 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout: legacy client fscid: -1 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout: No filesystems configured 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: mon.a@-1(???).osd e77 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: mon.a@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: mon.a@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T09:27:53.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: mon.a@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T09:27:53.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 ceph-mon[58326]: mon.a@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-09-17T09:27:53.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 bash[58306]: 88689aaf0f3f98c46cf717a6e8b89505e12255ebdcc0461e2dd3f3210ede43a4 2024-09-17T09:27:53.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:27:52 smithi029 systemd[1]: Started Ceph mon.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:54.276 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: ignoring --setuser ceph since I am not root 2024-09-17T09:27:54.277 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: ignoring --setgroup ceph since I am not root 2024-09-17T09:27:54.277 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-mgr[46749]: -- 172.21.15.103:0/2397913161 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560565914d00 con 0x5605658e8800 2024-09-17T09:27:54.277 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:54.275+0000 7f4578e54040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:27:54.277 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 2. 2024-09-17T09:27:54.277 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:54.277 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:27:54.277 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 podman[48086]: 2024-09-17 09:27:54.105366933 +0000 UTC m=+0.017760228 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:27:54.277 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 podman[48086]: 2024-09-17 09:27:54.224550076 +0000 UTC m=+0.136943355 container create bdc01a30b45028dd683d893943a2e23420b670d559f5e5042b6d5f446d0b9568 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:54.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:51.354187+0000 mgr.y (mgr.25054) 47 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-17T09:27:54.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:53.354544+0000 mgr.y (mgr.25054) 48 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-17T09:27:54.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:54.113694+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2024-09-17T09:27:54.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:54.115537+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T09:27:54.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:54.120114+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:27:54.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:54.120724+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-09-17T09:27:54.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:54.120750+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-17T09:27:54.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:54.121978+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e26: y(active, since 41s), standbys: x 2024-09-17T09:27:54.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:54.122508+0000 mon.a (mon.0) 7 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:27:54.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:54.122524+0000 mon.a (mon.0) 8 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:27:54.279 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:54.122532+0000 mon.a (mon.0) 9 : cluster [WRN] daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:27:54.279 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:54.122540+0000 mon.a (mon.0) 10 : cluster [WRN] daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:27:54.279 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:54.132847+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e27: y(active, since 41s), standbys: x 2024-09-17T09:27:54.279 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:54.132954+0000 mon.a (mon.0) 12 : audit [INF] from='mgr.25054 ' entity='' 2024-09-17T09:27:54.510 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: ignoring --setuser ceph since I am not root 2024-09-17T09:27:54.510 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: ignoring --setgroup ceph since I am not root 2024-09-17T09:27:54.510 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-mgr[53270]: -- 172.21.15.29:0/727733129 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bcfb7fcd00 con 0x55bcfb7d1000 2024-09-17T09:27:54.510 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:54.293+0000 7fe79c118040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T09:27:54.510 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:54.351+0000 7fe79c118040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:27:54.510 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:54.507+0000 7fe79c118040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:27:54.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:51.354187+0000 mgr.y (mgr.25054) 47 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-17T09:27:54.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:53.354544+0000 mgr.y (mgr.25054) 48 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 112 B/s rd, 0 op/s 2024-09-17T09:27:54.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:54.113694+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2024-09-17T09:27:54.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:54.115537+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T09:27:54.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:54.120114+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:27:54.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:54.120724+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-09-17T09:27:54.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:54.120750+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-17T09:27:54.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:54.121978+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e26: y(active, since 41s), standbys: x 2024-09-17T09:27:54.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:54.122508+0000 mon.a (mon.0) 7 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:27:54.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:54.122524+0000 mon.a (mon.0) 8 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:27:54.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:54.122532+0000 mon.a (mon.0) 9 : cluster [WRN] daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:27:54.512 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:54.122540+0000 mon.a (mon.0) 10 : cluster [WRN] daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:27:54.512 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:54.132847+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e27: y(active, since 41s), standbys: x 2024-09-17T09:27:54.512 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:54.132954+0000 mon.a (mon.0) 12 : audit [INF] from='mgr.25054 ' entity='' 2024-09-17T09:27:54.526 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:54.331+0000 7f4578e54040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T09:27:54.527 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:54.488+0000 7f4578e54040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T09:27:54.527 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 podman[48086]: 2024-09-17 09:27:54.476939454 +0000 UTC m=+0.389332745 container init bdc01a30b45028dd683d893943a2e23420b670d559f5e5042b6d5f446d0b9568 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:54.527 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 podman[48086]: 2024-09-17 09:27:54.480394874 +0000 UTC m=+0.392788161 container start bdc01a30b45028dd683d893943a2e23420b670d559f5e5042b6d5f446d0b9568 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:54.527 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.525Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:27:54.527 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.525Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:27:54.527 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.525Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:27:54.527 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.525Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:27:54.527 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.525Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:27:54.527 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.525Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:27:54.798 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:54.579+0000 7fe79c118040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:27:54.798 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:54.754+0000 7fe79c118040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:27:54.866 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:54.577+0000 7f4578e54040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T09:27:54.866 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:54.723+0000 7f4578e54040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:27:54.866 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:54.865+0000 7f4578e54040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:27:54.867 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.527Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:27:54.867 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.527Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:27:54.867 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.529Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:27:54.867 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.530Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=9 2024-09-17T09:27:54.867 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.538Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=9 2024-09-17T09:27:54.867 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.538Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=9 2024-09-17T09:27:54.867 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.538Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=9 2024-09-17T09:27:54.867 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.538Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=9 2024-09-17T09:27:54.867 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.539Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=9 2024-09-17T09:27:54.867 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.539Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=9 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.539Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=9 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.539Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=9 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.539Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=9 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.539Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=10.602563ms 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.541Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.541Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.541Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.542Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.542Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.542Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.542Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.542Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.542Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:27:54.868 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.542Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:27:54.869 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.542Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:27:54.869 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.542Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:27:54.869 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=info ts=2024-09-17T09:27:54.542Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:27:54.869 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[48125]: level=error ts=2024-09-17T09:27:54.542Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:27:54.869 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 bash[48086]: bdc01a30b45028dd683d893943a2e23420b670d559f5e5042b6d5f446d0b9568 2024-09-17T09:27:54.869 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:27:54.869 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:54 smithi103 podman[48142]: 2024-09-17 09:27:54.600790875 +0000 UTC m=+0.050264837 container died bdc01a30b45028dd683d893943a2e23420b670d559f5e5042b6d5f446d0b9568 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:55.178 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:54 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:54.905+0000 7fe79c118040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T09:27:55.494 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:55.237+0000 7fe79c118040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:27:55.494 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:55.300+0000 7fe79c118040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:27:55.494 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:55.492+0000 7fe79c118040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:27:55.505 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:55 smithi103 podman[48142]: 2024-09-17 09:27:55.22379555 +0000 UTC m=+0.673269500 container remove bdc01a30b45028dd683d893943a2e23420b670d559f5e5042b6d5f446d0b9568 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:27:55.505 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:55 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:27:55.505 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:27:55 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:27:55.506 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:55.177+0000 7f4578e54040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T09:27:55.506 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:55.241+0000 7f4578e54040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T09:27:55.506 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:55.423+0000 7f4578e54040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T09:27:56.392 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:55.134047+0000 mon.a (mon.0) 13 : cluster [DBG] mgrmap e28: y(active, since 42s), standbys: x 2024-09-17T09:27:56.392 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:56.035720+0000 mon.c (mon.1) 83 : audit [DBG] from='client.? 172.21.15.29:0/2267055492' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:56.505 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:56.291+0000 7f4578e54040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:27:56.505 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:56.356+0000 7f4578e54040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:27:56.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:55.134047+0000 mon.a (mon.0) 13 : cluster [DBG] mgrmap e28: y(active, since 42s), standbys: x 2024-09-17T09:27:56.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:56.035720+0000 mon.c (mon.1) 83 : audit [DBG] from='client.? 172.21.15.29:0/2267055492' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:27:56.764 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:56.485+0000 7fe79c118040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T09:27:56.765 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:56 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:56.555+0000 7fe79c118040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T09:27:57.319 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:56.387391+0000 mon.a (mon.0) 14 : audit [INF] from='client.44130 172.21.15.29:0/185205792' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: access denied 2024-09-17T09:27:57.320 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:57.063+0000 7f4578e54040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:27:57.320 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:57.178+0000 7f4578e54040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:27:57.327 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:56.387391+0000 mon.a (mon.0) 14 : audit [INF] from='client.44130 172.21.15.29:0/185205792' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: access denied 2024-09-17T09:27:57.591 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:57.318+0000 7f4578e54040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:27:57.591 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:57.379+0000 7f4578e54040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:27:57.591 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:57.523+0000 7f4578e54040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:27:57.591 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:57.590+0000 7f4578e54040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:27:57.597 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:57.325+0000 7fe79c118040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T09:27:57.598 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:57.446+0000 7fe79c118040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T09:27:57.868 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:57.649+0000 7f4578e54040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:27:57.869 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:57.803+0000 7f4578e54040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:27:57.869 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:57.867+0000 7f4578e54040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:27:57.889 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:57.595+0000 7fe79c118040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T09:27:57.889 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:57.659+0000 7fe79c118040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T09:27:57.889 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:57.816+0000 7fe79c118040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T09:27:57.890 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:57.887+0000 7fe79c118040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T09:27:58.134 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:57.934+0000 7f4578e54040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:27:58.134 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:58.015+0000 7f4578e54040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:27:58.134 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:58.073+0000 7f4578e54040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:27:58.134 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:58.133+0000 7f4578e54040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:27:58.187 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:57.949+0000 7fe79c118040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T09:27:58.187 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:58.116+0000 7fe79c118040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T09:27:58.187 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:58.184+0000 7fe79c118040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T09:27:58.466 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:58.254+0000 7fe79c118040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T09:27:58.466 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:58.340+0000 7fe79c118040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T09:27:58.466 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:58.402+0000 7fe79c118040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T09:27:58.466 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:58 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:58.464+0000 7fe79c118040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T09:27:59.255 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:27:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x[46745]: 2024-09-17T09:27:58.942+0000 7f4578e54040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:27:59.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:58.945125+0000 mon.a (mon.0) 15 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T09:27:59.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:58.945203+0000 mon.a (mon.0) 16 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:27:59.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:58.945986+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.? 172.21.15.103:0/3740810102' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:27:59.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:58.946724+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.? 172.21.15.103:0/3740810102' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:27:59.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:58.947360+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.? 172.21.15.103:0/3740810102' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:27:59.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:27:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:58.947606+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.? 172.21.15.103:0/3740810102' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:27:59.287 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:58.945125+0000 mon.a (mon.0) 15 : cluster [DBG] Standby manager daemon x restarted 2024-09-17T09:27:59.287 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:58.945203+0000 mon.a (mon.0) 16 : cluster [DBG] Standby manager daemon x started 2024-09-17T09:27:59.287 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:58.945986+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.? 172.21.15.103:0/3740810102' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T09:27:59.287 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:58.946724+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.? 172.21.15.103:0/3740810102' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T09:27:59.288 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:58.947360+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.? 172.21.15.103:0/3740810102' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T09:27:59.288 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:27:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:58.947606+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.? 172.21.15.103:0/3740810102' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T09:27:59.547 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:27:59 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:27:59.285+0000 7fe79c118040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T09:28:00.295 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:59.002008+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e29: y(active, since 45s), standbys: x 2024-09-17T09:28:00.296 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:59.288862+0000 mon.a (mon.0) 18 : cluster [INF] Active manager daemon y restarted 2024-09-17T09:28:00.296 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:59.289160+0000 mon.a (mon.0) 19 : cluster [INF] Activating manager daemon y 2024-09-17T09:28:00.296 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:59.294906+0000 mon.a (mon.0) 20 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-17T09:28:00.296 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:59.296232+0000 mon.a (mon.0) 21 : cluster [DBG] mgrmap e30: y(active, starting, since 0.00714194s), standbys: x 2024-09-17T09:28:00.296 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.302038+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:28:00.296 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.302186+0000 mon.c (mon.1) 85 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:28:00.296 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.302369+0000 mon.c (mon.1) 86 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:28:00.296 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.303256+0000 mon.c (mon.1) 87 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:28:00.296 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.303516+0000 mon.c (mon.1) 88 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:28:00.296 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.303783+0000 mon.c (mon.1) 89 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:28:00.296 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.304106+0000 mon.c (mon.1) 90 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:28:00.296 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.304409+0000 mon.c (mon.1) 91 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:28:00.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.304716+0000 mon.c (mon.1) 92 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:28:00.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.304985+0000 mon.c (mon.1) 93 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:28:00.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.305262+0000 mon.c (mon.1) 94 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:28:00.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.305532+0000 mon.c (mon.1) 95 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:28:00.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.305818+0000 mon.c (mon.1) 96 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:28:00.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.306114+0000 mon.c (mon.1) 97 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:28:00.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.306340+0000 mon.c (mon.1) 98 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:28:00.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.306832+0000 mon.c (mon.1) 99 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:28:00.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:27:59.313938+0000 mon.a (mon.0) 22 : cluster [INF] Manager daemon y is now available 2024-09-17T09:28:00.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.362545+0000 mon.c (mon.1) 100 : audit [INF] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:28:00.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.362971+0000 mon.a (mon.0) 23 : audit [INF] from='mgr.25264 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:28:00.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.380178+0000 mon.c (mon.1) 101 : audit [INF] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:28:00.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:27:59.380681+0000 mon.a (mon.0) 24 : audit [INF] from='mgr.25264 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:28:00.384 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:59.002008+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e29: y(active, since 45s), standbys: x 2024-09-17T09:28:00.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:59.288862+0000 mon.a (mon.0) 18 : cluster [INF] Active manager daemon y restarted 2024-09-17T09:28:00.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:59.289160+0000 mon.a (mon.0) 19 : cluster [INF] Activating manager daemon y 2024-09-17T09:28:00.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:59.294906+0000 mon.a (mon.0) 20 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-17T09:28:00.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:59.296232+0000 mon.a (mon.0) 21 : cluster [DBG] mgrmap e30: y(active, starting, since 0.00714194s), standbys: x 2024-09-17T09:28:00.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.302038+0000 mon.c (mon.1) 84 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:28:00.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.302186+0000 mon.c (mon.1) 85 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:28:00.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.302369+0000 mon.c (mon.1) 86 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:28:00.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.303256+0000 mon.c (mon.1) 87 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T09:28:00.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.303516+0000 mon.c (mon.1) 88 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T09:28:00.385 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.303783+0000 mon.c (mon.1) 89 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:28:00.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.304106+0000 mon.c (mon.1) 90 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:28:00.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.304409+0000 mon.c (mon.1) 91 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:28:00.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.304716+0000 mon.c (mon.1) 92 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:28:00.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.304985+0000 mon.c (mon.1) 93 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:28:00.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.305262+0000 mon.c (mon.1) 94 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:28:00.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.305532+0000 mon.c (mon.1) 95 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:28:00.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.305818+0000 mon.c (mon.1) 96 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:28:00.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.306114+0000 mon.c (mon.1) 97 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T09:28:00.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.306340+0000 mon.c (mon.1) 98 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T09:28:00.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.306832+0000 mon.c (mon.1) 99 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T09:28:00.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:27:59.313938+0000 mon.a (mon.0) 22 : cluster [INF] Manager daemon y is now available 2024-09-17T09:28:00.386 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.362545+0000 mon.c (mon.1) 100 : audit [INF] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:28:00.387 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.362971+0000 mon.a (mon.0) 23 : audit [INF] from='mgr.25264 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T09:28:00.387 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.380178+0000 mon.c (mon.1) 101 : audit [INF] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:28:00.387 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:27:59.380681+0000 mon.a (mon.0) 24 : audit [INF] from='mgr.25264 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T09:28:00.547 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:28:00.299+0000 7fe73b7e7640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-17T09:28:01.513 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:28:00.299734+0000 mon.a (mon.0) 25 : cluster [DBG] mgrmap e31: y(active, since 1.01065s), standbys: x 2024-09-17T09:28:01.513 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:28:00.310185+0000 mgr.y (mgr.25264) 1 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:01.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:00.299734+0000 mon.a (mon.0) 25 : cluster [DBG] mgrmap e31: y(active, since 1.01065s), standbys: x 2024-09-17T09:28:01.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:00.310185+0000 mgr.y (mgr.25264) 1 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:02.632 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:01.294633+0000 mon.a (mon.0) 26 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T09:28:02.632 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:01.294673+0000 mon.a (mon.0) 27 : cluster [INF] Cluster is now healthy 2024-09-17T09:28:02.632 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:01.305112+0000 mon.a (mon.0) 28 : cluster [DBG] mgrmap e32: y(active, since 2s), standbys: x 2024-09-17T09:28:02.632 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:28:02.164391+0000 mgr.y (mgr.25264) 3 : cephadm [INF] [17/Sep/2024:09:28:02] ENGINE Bus STARTING 2024-09-17T09:28:02.709 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:28:01.294633+0000 mon.a (mon.0) 26 : cluster [INF] Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T09:28:02.709 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:28:01.294673+0000 mon.a (mon.0) 27 : cluster [INF] Cluster is now healthy 2024-09-17T09:28:02.709 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:28:01.305112+0000 mon.a (mon.0) 28 : cluster [DBG] mgrmap e32: y(active, since 2s), standbys: x 2024-09-17T09:28:02.710 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:28:02.164391+0000 mgr.y (mgr.25264) 3 : cephadm [INF] [17/Sep/2024:09:28:02] ENGINE Bus STARTING 2024-09-17T09:28:02.710 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:28:02.384+0000 7fe6f868f640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:09:28:02] ENGINE Error in HTTPServer.serve 2024-09-17T09:28:02.710 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: Traceback (most recent call last): 2024-09-17T09:28:02.710 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:28:02.710 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: self._connections.run(self.expiration_interval) 2024-09-17T09:28:02.710 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:28:02.710 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: self._run(expiration_interval) 2024-09-17T09:28:02.710 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:28:02.710 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:28:02.710 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:28:02.710 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:28:02.711 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:28:02.711 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: s = self.context.wrap_socket( 2024-09-17T09:28:02.711 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:28:02.711 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: return self.sslsocket_class._create( 2024-09-17T09:28:02.711 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:28:02.711 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: self.do_handshake() 2024-09-17T09:28:02.711 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:28:02.711 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: self._sslobj.do_handshake() 2024-09-17T09:28:02.711 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:28:02.711 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:02 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:28:03.641 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:28:02.266039+0000 mgr.y (mgr.25264) 4 : cephadm [INF] [17/Sep/2024:09:28:02] ENGINE Serving on http://172.21.15.29:8765 2024-09-17T09:28:03.641 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:28:02.384986+0000 mgr.y (mgr.25264) 5 : cephadm [INF] [17/Sep/2024:09:28:02] ENGINE Serving on https://172.21.15.29:7150 2024-09-17T09:28:03.641 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:28:02.385096+0000 mgr.y (mgr.25264) 6 : cephadm [INF] [17/Sep/2024:09:28:02] ENGINE Bus STARTED 2024-09-17T09:28:03.641 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:28:02.386251+0000 mgr.y (mgr.25264) 7 : cephadm [ERR] [17/Sep/2024:09:28:02] ENGINE Error in HTTPServer.serve 2024-09-17T09:28:03.641 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: Traceback (most recent call last): 2024-09-17T09:28:03.641 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:28:03.641 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: self._connections.run(self.expiration_interval) 2024-09-17T09:28:03.641 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:28:03.642 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: self._run(expiration_interval) 2024-09-17T09:28:03.642 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:28:03.642 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:28:03.642 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:28:03.642 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:28:03.642 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:28:03.642 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: s = self.context.wrap_socket( 2024-09-17T09:28:03.642 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:28:03.642 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: return self.sslsocket_class._create( 2024-09-17T09:28:03.642 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:28:03.642 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: self.do_handshake() 2024-09-17T09:28:03.642 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:28:03.643 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: self._sslobj.do_handshake() 2024-09-17T09:28:03.643 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:28:03.643 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: 2024-09-17T09:28:03.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:28:02.266039+0000 mgr.y (mgr.25264) 4 : cephadm [INF] [17/Sep/2024:09:28:02] ENGINE Serving on http://172.21.15.29:8765 2024-09-17T09:28:03.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:28:02.384986+0000 mgr.y (mgr.25264) 5 : cephadm [INF] [17/Sep/2024:09:28:02] ENGINE Serving on https://172.21.15.29:7150 2024-09-17T09:28:03.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:28:02.385096+0000 mgr.y (mgr.25264) 6 : cephadm [INF] [17/Sep/2024:09:28:02] ENGINE Bus STARTED 2024-09-17T09:28:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:28:02.386251+0000 mgr.y (mgr.25264) 7 : cephadm [ERR] [17/Sep/2024:09:28:02] ENGINE Error in HTTPServer.serve 2024-09-17T09:28:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: Traceback (most recent call last): 2024-09-17T09:28:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:28:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: self._connections.run(self.expiration_interval) 2024-09-17T09:28:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:28:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: self._run(expiration_interval) 2024-09-17T09:28:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:28:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:28:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:28:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:28:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:28:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: s = self.context.wrap_socket( 2024-09-17T09:28:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:28:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: return self.sslsocket_class._create( 2024-09-17T09:28:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:28:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: self.do_handshake() 2024-09-17T09:28:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:28:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: self._sslobj.do_handshake() 2024-09-17T09:28:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:28:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: 2024-09-17T09:28:04.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:03.306702+0000 mgr.y (mgr.25264) 8 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:04.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:03.311089+0000 mon.a (mon.0) 29 : cluster [DBG] mgrmap e33: y(active, since 4s), standbys: x 2024-09-17T09:28:04.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:04.249926+0000 mon.a (mon.0) 30 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:04.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:04.256282+0000 mon.a (mon.0) 31 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:04.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:28:03.306702+0000 mgr.y (mgr.25264) 8 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:04.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:28:03.311089+0000 mon.a (mon.0) 29 : cluster [DBG] mgrmap e33: y(active, since 4s), standbys: x 2024-09-17T09:28:04.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:04.249926+0000 mon.a (mon.0) 30 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:04.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:04.256282+0000 mon.a (mon.0) 31 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:05.420 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:05 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 3. 2024-09-17T09:28:05.421 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:05 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:05.421 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:05 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:28:05.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:04.570088+0000 mon.a (mon.0) 32 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:04.575709+0000 mon.a (mon.0) 33 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:04.769722+0000 mon.a (mon.0) 34 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:04.777670+0000 mon.a (mon.0) 35 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:05.110506+0000 mon.a (mon.0) 36 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:05.115781+0000 mon.a (mon.0) 37 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:28:06.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:04.570088+0000 mon.a (mon.0) 32 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:06.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:04.575709+0000 mon.a (mon.0) 33 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:06.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:04.769722+0000 mon.a (mon.0) 34 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:06.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:04.777670+0000 mon.a (mon.0) 35 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:06.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:05.110506+0000 mon.a (mon.0) 36 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:06.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:05.115781+0000 mon.a (mon.0) 37 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:06.447 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:28:06.448 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (103s) 1s ago 6m 11.2M - 0.20.0 0881eb8f169f 7bd158a6d65c 2024-09-17T09:28:06.448 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (73s) 2s ago 6m 28.8M - 6.7.4 557c83e11646 e361816f7c14 2024-09-17T09:28:06.448 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 unknown 1s ago 5m - - 2024-09-17T09:28:06.448 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (29s) 2s ago 10m 434M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:28:06.448 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (66s) 1s ago 11m 456M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:28:06.448 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (14s) 1s ago 11m 28.5M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:28:06.448 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (10m) 2s ago 10m 45.2M 2048M 16.2.0 24ecd6d5f14c 203d1f26d7b8 2024-09-17T09:28:06.448 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (10m) 1s ago 10m 44.4M 2048M 16.2.0 24ecd6d5f14c aa05f3e5656a 2024-09-17T09:28:06.448 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (112s) 1s ago 6m 15.0M - 0.18.1 e5a616e4b9cf 34bfa263169f 2024-09-17T09:28:06.448 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (85s) 2s ago 6m 6777k - 0.18.1 e5a616e4b9cf 1adf1b5baad1 2024-09-17T09:28:06.448 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (9m) 1s ago 9m 48.0M 4096M 16.2.0 24ecd6d5f14c 991731ce08b5 2024-09-17T09:28:06.448 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (9m) 1s ago 9m 46.1M 4096M 16.2.0 24ecd6d5f14c d5bce63fadc8 2024-09-17T09:28:06.449 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (9m) 1s ago 9m 43.3M 4096M 16.2.0 24ecd6d5f14c e9678d2f1685 2024-09-17T09:28:06.449 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (8m) 1s ago 8m 49.5M 4096M 16.2.0 24ecd6d5f14c dcf7122a7d99 2024-09-17T09:28:06.449 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (8m) 2s ago 8m 49.0M 4096M 16.2.0 24ecd6d5f14c 048c13025d32 2024-09-17T09:28:06.449 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (8m) 2s ago 8m 47.5M 4096M 16.2.0 24ecd6d5f14c 10a64919fbc9 2024-09-17T09:28:06.449 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (8m) 2s ago 8m 46.1M 4096M 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:28:06.449 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (7m) 2s ago 7m 52.3M 4096M 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:28:06.449 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 unknown 2s ago 7m - - 2024-09-17T09:28:06.449 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (5m) 1s ago 5m 40.3M - 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:28:06.449 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (5m) 2s ago 5m 40.4M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:28:06.449 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (5m) 1s ago 5m 38.4M - 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:28:06.449 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (5m) 2s ago 5m 37.4M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:28:06.490 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 podman[49174]: 2024-09-17 09:28:06.231176101 +0000 UTC m=+0.446641916 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:28:06.490 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 podman[49174]: 2024-09-17 09:28:06.366820832 +0000 UTC m=+0.582286647 container create c12ace40963e739a8ee8d8621172a543aea09e5d1e21fa4c49a4963524a5c378 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:06.846 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:28:06.846 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:28:06.846 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2, 2024-09-17T09:28:06.846 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T09:28:06.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:28:06.846 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:28:06.847 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:28:06.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:28:06.847 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:28:06.847 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T09:28:06.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:28:06.847 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": {}, 2024-09-17T09:28:06.847 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:28:06.847 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:28:06.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:28:06.847 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:28:06.847 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 14, 2024-09-17T09:28:06.847 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:28:06.847 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:28:06.848 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:28:07.118 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 podman[49174]: 2024-09-17 09:28:06.813308758 +0000 UTC m=+1.028774577 container init c12ace40963e739a8ee8d8621172a543aea09e5d1e21fa4c49a4963524a5c378 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:07.118 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 podman[49174]: 2024-09-17 09:28:06.816090243 +0000 UTC m=+1.031556053 container start c12ace40963e739a8ee8d8621172a543aea09e5d1e21fa4c49a4963524a5c378 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:07.118 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.846Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.846Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.846Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.846Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.846Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.846Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.847Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.847Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.849Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.849Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=10 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.857Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=10 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.857Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=10 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.857Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=10 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.857Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=10 2024-09-17T09:28:07.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.857Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=10 2024-09-17T09:28:07.120 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.857Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=10 2024-09-17T09:28:07.120 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.858Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=10 2024-09-17T09:28:07.120 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.858Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=10 2024-09-17T09:28:07.120 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.858Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=10 2024-09-17T09:28:07.120 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.858Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=10 2024-09-17T09:28:07.120 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.858Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=9.011541ms 2024-09-17T09:28:07.120 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.861Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:28:07.120 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.861Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:28:07.121 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.861Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:28:07.121 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.861Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:28:07.121 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.862Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:28:07.121 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.862Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:28:07.121 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.862Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:28:07.121 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.862Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:28:07.121 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.862Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:28:07.121 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.862Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:28:07.121 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.862Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:28:07.121 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.862Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:28:07.121 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=info ts=2024-09-17T09:28:06.862Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:28:07.121 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49214]: level=error ts=2024-09-17T09:28:06.862Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:28:07.188 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:28:07.188 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:28:07.188 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:28:07.188 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:28:07.188 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [ 2024-09-17T09:28:07.188 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr" 2024-09-17T09:28:07.189 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:28:07.189 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "3/23 daemons upgraded", 2024-09-17T09:28:07.189 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "", 2024-09-17T09:28:07.189 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:28:07.189 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:28:07.505 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:07 smithi103 bash[49174]: c12ace40963e739a8ee8d8621172a543aea09e5d1e21fa4c49a4963524a5c378 2024-09-17T09:28:07.505 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:07 smithi103 podman[49236]: 2024-09-17 09:28:07.124343645 +0000 UTC m=+0.254701986 container died c12ace40963e739a8ee8d8621172a543aea09e5d1e21fa4c49a4963524a5c378 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:07.506 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:07 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:07.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:05.307501+0000 mgr.y (mgr.25264) 9 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-17T09:28:07.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:05.749655+0000 mgr.y (mgr.25264) 10 : audit [DBG] from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:07.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:06.089587+0000 mgr.y (mgr.25264) 11 : audit [DBG] from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:07.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:06.845568+0000 mon.c (mon.1) 102 : audit [DBG] from='client.? 172.21.15.29:0/1781687977' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:07.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:28:05.307501+0000 mgr.y (mgr.25264) 9 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-17T09:28:07.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:05.749655+0000 mgr.y (mgr.25264) 10 : audit [DBG] from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:07.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:06.089587+0000 mgr.y (mgr.25264) 11 : audit [DBG] from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:07.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:07 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:06.845568+0000 mon.c (mon.1) 102 : audit [DBG] from='client.? 172.21.15.29:0/1781687977' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:07.586 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_OK 2024-09-17T09:28:08.123 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:07 smithi103 podman[49236]: 2024-09-17 09:28:07.845093778 +0000 UTC m=+0.975452144 container remove c12ace40963e739a8ee8d8621172a543aea09e5d1e21fa4c49a4963524a5c378 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:08.123 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:07 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:28:08.123 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:07 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:28:08.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:06.438488+0000 mgr.y (mgr.25264) 12 : audit [DBG] from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:08.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:07.186764+0000 mgr.y (mgr.25264) 13 : audit [DBG] from='client.25342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:08.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:07.585853+0000 mon.a (mon.0) 38 : audit [DBG] from='client.44190 172.21.15.29:0/197166824' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:28:08.491 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:06.438488+0000 mgr.y (mgr.25264) 12 : audit [DBG] from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:08.492 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:07.186764+0000 mgr.y (mgr.25264) 13 : audit [DBG] from='client.25342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:08.492 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:07.585853+0000 mon.a (mon.0) 38 : audit [DBG] from='client.44190 172.21.15.29:0/197166824' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:28:09.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:28:07.307986+0000 mgr.y (mgr.25264) 14 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T09:28:09.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:08.518847+0000 mon.a (mon.0) 39 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:09.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:08.526089+0000 mon.a (mon.0) 40 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:09.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:08.527729+0000 mon.c (mon.1) 103 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:09.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:08.528449+0000 mon.c (mon.1) 104 : audit [INF] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:28:09.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:08.532783+0000 mon.a (mon.0) 41 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:09.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:08.645168+0000 mon.c (mon.1) 105 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:28:09.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:08.647604+0000 mon.c (mon.1) 106 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:09.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:08.648567+0000 mon.c (mon.1) 107 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T09:28:09.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:09 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:08.649185+0000 mon.c (mon.1) 108 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T09:28:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:07.307986+0000 mgr.y (mgr.25264) 14 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T09:28:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:08.518847+0000 mon.a (mon.0) 39 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:08.526089+0000 mon.a (mon.0) 40 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:08.527729+0000 mon.c (mon.1) 103 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:08.528449+0000 mon.c (mon.1) 104 : audit [INF] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:28:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:08.532783+0000 mon.a (mon.0) 41 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:08.645168+0000 mon.c (mon.1) 105 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:28:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:08.647604+0000 mon.c (mon.1) 106 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:08.648567+0000 mon.c (mon.1) 107 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T09:28:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:08.649185+0000 mon.c (mon.1) 108 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T09:28:10.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:28:08.529022+0000 mgr.y (mgr.25264) 15 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:10.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:28:08.529497+0000 mgr.y (mgr.25264) 16 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-17T09:28:10.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:28:08.649642+0000 mgr.y (mgr.25264) 17 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2024-09-17T09:28:10.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:09.286903+0000 mon.c (mon.1) 109 : audit [DBG] from='client.? 172.21.15.29:0/871367148' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:10.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:28:09.524399+0000 mon.a (mon.0) 42 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:28:10.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:09.629138+0000 mon.a (mon.0) 43 : audit [INF] from='client.44214 172.21.15.29:0/552312000' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]: access denied 2024-09-17T09:28:10.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:10.060082+0000 mon.a (mon.0) 44 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:10.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:10.061454+0000 mon.c (mon.1) 110 : audit [INF] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:10.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:10.062297+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.25264 ' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:10.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: audit 2024-09-17T09:28:10.062983+0000 mon.c (mon.1) 111 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:10.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:28:08.529022+0000 mgr.y (mgr.25264) 15 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:10.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:08.529497+0000 mgr.y (mgr.25264) 16 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-17T09:28:10.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:28:08.649642+0000 mgr.y (mgr.25264) 17 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2024-09-17T09:28:10.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:09.286903+0000 mon.c (mon.1) 109 : audit [DBG] from='client.? 172.21.15.29:0/871367148' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:10.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:09.524399+0000 mon.a (mon.0) 42 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:28:10.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:09.629138+0000 mon.a (mon.0) 43 : audit [INF] from='client.44214 172.21.15.29:0/552312000' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]: access denied 2024-09-17T09:28:10.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:10.060082+0000 mon.a (mon.0) 44 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:10.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:10.061454+0000 mon.c (mon.1) 110 : audit [INF] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:10.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:10.062297+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.25264 ' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:10.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:10.062983+0000 mon.c (mon.1) 111 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:11.485 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:28:10.055356+0000 mgr.y (mgr.25264) 18 : cephadm [INF] Upgrade: Updating mon.c 2024-09-17T09:28:11.485 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cephadm 2024-09-17T09:28:10.063930+0000 mgr.y (mgr.25264) 19 : cephadm [INF] Deploying daemon mon.c on smithi029 2024-09-17T09:28:11.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:28:10.055356+0000 mgr.y (mgr.25264) 18 : cephadm [INF] Upgrade: Updating mon.c 2024-09-17T09:28:11.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:28:10.063930+0000 mgr.y (mgr.25264) 19 : cephadm [INF] Deploying daemon mon.c on smithi029 2024-09-17T09:28:12.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:10.530200+0000 mgr.y (mgr.25264) 20 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:28:12.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: cluster 2024-09-17T09:28:10.530200+0000 mgr.y (mgr.25264) 20 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:28:13.451 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:13 smithi029 systemd[1]: Stopping Ceph mon.c for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:28:13.452 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:28:13.368+0000 7f592d06d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-17T09:28:13.452 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c[26946]: debug 2024-09-17T09:28:13.368+0000 7f592d06d700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-17T09:28:13.774 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:13 smithi029 podman[60317]: 2024-09-17 09:28:13.45097971 +0000 UTC m=+0.219653602 container died aa05f3e5656a941e7d8d4d0a8835726559f8a8f54bdd8b7bdc2d55180184605d (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, ceph=True) 2024-09-17T09:28:14.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:13 smithi029 podman[60317]: 2024-09-17 09:28:13.872795716 +0000 UTC m=+0.641469608 container cleanup aa05f3e5656a941e7d8d4d0a8835726559f8a8f54bdd8b7bdc2d55180184605d (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20201204, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0) 2024-09-17T09:28:14.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:13 smithi029 bash[60317]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c 2024-09-17T09:28:14.433 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:14 smithi029 podman[60330]: 2024-09-17 09:28:14.08337578 +0000 UTC m=+0.636718620 container remove aa05f3e5656a941e7d8d4d0a8835726559f8a8f54bdd8b7bdc2d55180184605d (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.c, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557) 2024-09-17T09:28:14.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:14 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-17T09:28:14.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:14 smithi029 systemd[1]: Stopped Ceph mon.c for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:14.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:14 smithi029 systemd[1]: Starting Ceph mon.c for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:28:15.184 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:14 smithi029 podman[60470]: 2024-09-17 09:28:14.77588154 +0000 UTC m=+0.016168980 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:28:15.184 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:14 smithi029 podman[60470]: 2024-09-17 09:28:14.903963505 +0000 UTC m=+0.144250929 container create 0d3d1a65e2c90bcd7059d6fffa435afdbbb8f998f6048dd6f5c04723ffdc3154 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-c, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-17T09:28:15.184 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 podman[60470]: 2024-09-17 09:28:15.183342893 +0000 UTC m=+0.423630336 container init 0d3d1a65e2c90bcd7059d6fffa435afdbbb8f998f6048dd6f5c04723ffdc3154 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-17T09:28:15.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 podman[60470]: 2024-09-17 09:28:15.186942118 +0000 UTC m=+0.427229542 container start 0d3d1a65e2c90bcd7059d6fffa435afdbbb8f998f6048dd6f5c04723ffdc3154 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-17T09:28:15.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T09:28:15.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process ceph-mon, pid 2 2024-09-17T09:28:15.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: pidfile_write: ignore empty --pid-file 2024-09-17T09:28:15.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: load: jerasure load: lrc 2024-09-17T09:28:15.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: RocksDB version: 7.9.2 2024-09-17T09:28:15.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Git sha 0 2024-09-17T09:28:15.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Compile date 2024-09-16 15:15:13 2024-09-17T09:28:15.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: DB SUMMARY 2024-09-17T09:28:15.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: DB Session ID: IW0PKKRG6ET6233NWOIN 2024-09-17T09:28:15.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: CURRENT file: CURRENT 2024-09-17T09:28:15.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T09:28:15.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: MANIFEST file: MANIFEST-000005 size: 896 Bytes 2024-09-17T09:28:15.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000020.sst 2024-09-17T09:28:15.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000018.log size: 5952503 ; 2024-09-17T09:28:15.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.error_if_exists: 0 2024-09-17T09:28:15.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.create_if_missing: 0 2024-09-17T09:28:15.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.paranoid_checks: 1 2024-09-17T09:28:15.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T09:28:15.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T09:28:15.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T09:28:15.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.env: 0x55c64b82abc0 2024-09-17T09:28:15.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T09:28:15.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.info_log: 0x55c64da16be0 2024-09-17T09:28:15.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T09:28:15.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.statistics: (nil) 2024-09-17T09:28:15.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.use_fsync: 0 2024-09-17T09:28:15.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_log_file_size: 0 2024-09-17T09:28:15.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T09:28:15.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T09:28:15.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T09:28:15.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T09:28:15.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.allow_fallocate: 1 2024-09-17T09:28:15.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T09:28:15.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T09:28:15.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.use_direct_reads: 0 2024-09-17T09:28:15.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T09:28:15.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T09:28:15.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.db_log_dir: 2024-09-17T09:28:15.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.wal_dir: 2024-09-17T09:28:15.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T09:28:15.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T09:28:15.555 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T09:28:15.555 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T09:28:15.555 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T09:28:15.555 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T09:28:15.555 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T09:28:15.555 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T09:28:15.555 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.write_buffer_manager: 0x55c64da24b40 2024-09-17T09:28:15.555 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T09:28:15.556 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T09:28:15.556 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T09:28:15.556 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T09:28:15.556 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T09:28:15.556 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T09:28:15.556 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T09:28:15.556 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T09:28:15.557 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.unordered_write: 0 2024-09-17T09:28:15.557 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T09:28:15.557 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T09:28:15.557 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T09:28:15.557 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T09:28:15.557 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.row_cache: None 2024-09-17T09:28:15.557 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.wal_filter: None 2024-09-17T09:28:15.558 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T09:28:15.558 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T09:28:15.558 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.two_write_queues: 0 2024-09-17T09:28:15.558 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T09:28:15.558 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.wal_compression: 0 2024-09-17T09:28:15.558 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.atomic_flush: 0 2024-09-17T09:28:15.558 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T09:28:15.558 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T09:28:15.559 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T09:28:15.560 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.log_readahead_size: 0 2024-09-17T09:28:15.560 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T09:28:15.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T09:28:15.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T09:28:15.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T09:28:15.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T09:28:15.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T09:28:15.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T09:28:15.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_background_jobs: 2 2024-09-17T09:28:15.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_background_compactions: -1 2024-09-17T09:28:15.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_subcompactions: 1 2024-09-17T09:28:15.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T09:28:15.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T09:28:15.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T09:28:15.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T09:28:15.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T09:28:15.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T09:28:15.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T09:28:15.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T09:28:15.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_open_files: -1 2024-09-17T09:28:15.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T09:28:15.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T09:28:15.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T09:28:15.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T09:28:15.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_background_flushes: -1 2024-09-17T09:28:15.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Compression algorithms supported: 2024-09-17T09:28:15.564 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: kZSTD supported: 0 2024-09-17T09:28:15.564 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: kXpressCompression supported: 0 2024-09-17T09:28:15.564 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: kBZip2Compression supported: 0 2024-09-17T09:28:15.564 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T09:28:15.564 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: kLZ4Compression supported: 1 2024-09-17T09:28:15.564 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: kZlibCompression supported: 1 2024-09-17T09:28:15.564 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T09:28:15.565 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: kSnappyCompression supported: 1 2024-09-17T09:28:15.565 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T09:28:15.565 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T09:28:15.565 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-17T09:28:15.565 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T09:28:15.565 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T09:28:15.565 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.merge_operator: 2024-09-17T09:28:15.565 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compaction_filter: None 2024-09-17T09:28:15.566 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T09:28:15.566 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T09:28:15.566 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T09:28:15.566 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T09:28:15.566 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c64da167e0) 2024-09-17T09:28:15.566 INFO:journalctl@ceph.mon.c.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-09-17T09:28:15.566 INFO:journalctl@ceph.mon.c.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T09:28:15.566 INFO:journalctl@ceph.mon.c.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T09:28:15.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-09-17T09:28:15.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: index_type: 0 2024-09-17T09:28:15.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: data_block_index_type: 0 2024-09-17T09:28:15.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: index_shortening: 1 2024-09-17T09:28:15.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T09:28:15.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: checksum: 4 2024-09-17T09:28:15.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: no_block_cache: 0 2024-09-17T09:28:15.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_cache: 0x55c64da0b350 2024-09-17T09:28:15.568 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-09-17T09:28:15.568 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_cache_options: 2024-09-17T09:28:15.568 INFO:journalctl@ceph.mon.c.smithi029.stdout: capacity : 536870912 2024-09-17T09:28:15.568 INFO:journalctl@ceph.mon.c.smithi029.stdout: num_shard_bits : 4 2024-09-17T09:28:15.568 INFO:journalctl@ceph.mon.c.smithi029.stdout: strict_capacity_limit : 0 2024-09-17T09:28:15.568 INFO:journalctl@ceph.mon.c.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-09-17T09:28:15.568 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_cache_compressed: (nil) 2024-09-17T09:28:15.568 INFO:journalctl@ceph.mon.c.smithi029.stdout: persistent_cache: (nil) 2024-09-17T09:28:15.569 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_size: 4096 2024-09-17T09:28:15.569 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_size_deviation: 10 2024-09-17T09:28:15.569 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_restart_interval: 16 2024-09-17T09:28:15.569 INFO:journalctl@ceph.mon.c.smithi029.stdout: index_block_restart_interval: 1 2024-09-17T09:28:15.569 INFO:journalctl@ceph.mon.c.smithi029.stdout: metadata_block_size: 4096 2024-09-17T09:28:15.569 INFO:journalctl@ceph.mon.c.smithi029.stdout: partition_filters: 0 2024-09-17T09:28:15.569 INFO:journalctl@ceph.mon.c.smithi029.stdout: use_delta_encoding: 1 2024-09-17T09:28:15.569 INFO:journalctl@ceph.mon.c.smithi029.stdout: filter_policy: bloomfilter 2024-09-17T09:28:15.570 INFO:journalctl@ceph.mon.c.smithi029.stdout: whole_key_filtering: 1 2024-09-17T09:28:15.570 INFO:journalctl@ceph.mon.c.smithi029.stdout: verify_compression: 0 2024-09-17T09:28:15.570 INFO:journalctl@ceph.mon.c.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-09-17T09:28:15.570 INFO:journalctl@ceph.mon.c.smithi029.stdout: format_version: 5 2024-09-17T09:28:15.570 INFO:journalctl@ceph.mon.c.smithi029.stdout: enable_index_compression: 1 2024-09-17T09:28:15.570 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_align: 0 2024-09-17T09:28:15.570 INFO:journalctl@ceph.mon.c.smithi029.stdout: max_auto_readahead_size: 262144 2024-09-17T09:28:15.570 INFO:journalctl@ceph.mon.c.smithi029.stdout: prepopulate_block_cache: 0 2024-09-17T09:28:15.571 INFO:journalctl@ceph.mon.c.smithi029.stdout: initial_auto_readahead_size: 8192 2024-09-17T09:28:15.571 INFO:journalctl@ceph.mon.c.smithi029.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T09:28:15.571 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T09:28:15.571 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T09:28:15.571 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compression: NoCompression 2024-09-17T09:28:15.571 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T09:28:15.571 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T09:28:15.571 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T09:28:15.572 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.num_levels: 7 2024-09-17T09:28:15.572 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T09:28:15.572 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T09:28:15.572 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T09:28:15.572 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T09:28:15.572 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T09:28:15.572 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T09:28:15.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T09:28:15.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:28:15.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T09:28:15.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T09:28:15.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T09:28:15.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T09:28:15.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T09:28:15.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T09:28:15.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T09:28:15.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T09:28:15.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:28:15.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T09:28:15.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T09:28:15.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T09:28:15.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T09:28:15.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T09:28:15.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T09:28:15.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T09:28:15.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T09:28:15.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T09:28:15.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T09:28:15.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T09:28:15.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T09:28:15.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T09:28:15.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T09:28:15.576 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T09:28:15.576 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T09:28:15.576 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T09:28:15.576 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T09:28:15.576 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T09:28:15.576 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T09:28:15.576 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T09:28:15.577 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T09:28:15.577 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T09:28:15.577 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T09:28:15.577 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T09:28:15.577 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T09:28:15.577 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T09:28:15.577 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T09:28:15.577 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T09:28:15.577 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T09:28:15.578 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T09:28:15.578 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T09:28:15.578 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T09:28:15.578 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T09:28:15.578 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T09:28:15.578 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T09:28:15.578 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T09:28:15.578 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.inplace_update_support: 0 2024-09-17T09:28:15.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T09:28:15.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T09:28:15.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T09:28:15.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T09:28:15.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.bloom_locality: 0 2024-09-17T09:28:15.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.max_successive_merges: 0 2024-09-17T09:28:15.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T09:28:15.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T09:28:15.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T09:28:15.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T09:28:15.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.ttl: 2592000 2024-09-17T09:28:15.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T09:28:15.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T09:28:15.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T09:28:15.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.enable_blob_files: false 2024-09-17T09:28:15.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.min_blob_size: 0 2024-09-17T09:28:15.581 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T09:28:15.581 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T09:28:15.581 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T09:28:15.581 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T09:28:15.581 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T09:28:15.581 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T09:28:15.581 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T09:28:15.581 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T09:28:15.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 20.sst 2024-09-17T09:28:15.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T09:28:15.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 22, last_sequence is 12120, log_number is 18,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T09:28:15.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 18 2024-09-17T09:28:15.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7f696dbc-5763-47f1-b64e-bed87988f308 2024-09-17T09:28:15.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726565295214070, "job": 1, "event": "recovery_started", "wal_files": [18]} 2024-09-17T09:28:15.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #18 mode 2 2024-09-17T09:28:15.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726565295227458, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 23, "file_size": 3869734, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12117, "largest_seqno": 12741, "table_properties": {"data_size": 3865371, "index_size": 2743, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 4652, "raw_average_key_size": 21, "raw_value_size": 3859149, "raw_average_value_size": 18118, "num_data_blocks": 123, "num_entries": 213, "num_filter_entries": 213, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726565295, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7f696dbc-5763-47f1-b64e-bed87988f308", "db_session_id": "IW0PKKRG6ET6233NWOIN", "orig_file_number": 23, "seqno_to_time_mapping": "N/A"}} 2024-09-17T09:28:15.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726565295227561, "job": 1, "event": "recovery_finished"} 2024-09-17T09:28:15.583 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [db/version_set.cc:5047] Creating manifest 25 2024-09-17T09:28:15.583 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T09:28:15.583 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000018.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T09:28:15.583 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c64da36e00 2024-09-17T09:28:15.583 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: DB pointer 0x55c64db2a000 2024-09-17T09:28:15.583 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T09:28:15.583 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T09:28:15.583 INFO:journalctl@ceph.mon.c.smithi029.stdout: ** DB Stats ** 2024-09-17T09:28:15.584 INFO:journalctl@ceph.mon.c.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:28:15.584 INFO:journalctl@ceph.mon.c.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T09:28:15.584 INFO:journalctl@ceph.mon.c.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:28:15.584 INFO:journalctl@ceph.mon.c.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:28:15.584 INFO:journalctl@ceph.mon.c.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T09:28:15.584 INFO:journalctl@ceph.mon.c.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:28:15.584 INFO:journalctl@ceph.mon.c.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:28:15.585 INFO:journalctl@ceph.mon.c.smithi029.stdout: 2024-09-17T09:28:15.585 INFO:journalctl@ceph.mon.c.smithi029.stdout: ** Compaction Stats [default] ** 2024-09-17T09:28:15.585 INFO:journalctl@ceph.mon.c.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T09:28:15.585 INFO:journalctl@ceph.mon.c.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T09:28:15.585 INFO:journalctl@ceph.mon.c.smithi029.stdout: L0 1/0 3.69 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 678.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-09-17T09:28:15.585 INFO:journalctl@ceph.mon.c.smithi029.stdout: L6 1/0 24.60 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-17T09:28:15.585 INFO:journalctl@ceph.mon.c.smithi029.stdout: Sum 2/0 28.29 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 678.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-09-17T09:28:15.585 INFO:journalctl@ceph.mon.c.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 678.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-09-17T09:28:15.586 INFO:journalctl@ceph.mon.c.smithi029.stdout: 2024-09-17T09:28:15.586 INFO:journalctl@ceph.mon.c.smithi029.stdout: ** Compaction Stats [default] ** 2024-09-17T09:28:15.586 INFO:journalctl@ceph.mon.c.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T09:28:15.586 INFO:journalctl@ceph.mon.c.smithi029.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:28:15.586 INFO:journalctl@ceph.mon.c.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 678.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-09-17T09:28:15.586 INFO:journalctl@ceph.mon.c.smithi029.stdout: 2024-09-17T09:28:15.586 INFO:journalctl@ceph.mon.c.smithi029.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T09:28:15.587 INFO:journalctl@ceph.mon.c.smithi029.stdout: 2024-09-17T09:28:15.587 INFO:journalctl@ceph.mon.c.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:28:15.587 INFO:journalctl@ceph.mon.c.smithi029.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-09-17T09:28:15.587 INFO:journalctl@ceph.mon.c.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:28:15.587 INFO:journalctl@ceph.mon.c.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:28:15.587 INFO:journalctl@ceph.mon.c.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:28:15.587 INFO:journalctl@ceph.mon.c.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:28:15.588 INFO:journalctl@ceph.mon.c.smithi029.stdout: Cumulative compaction: 0.00 GB write, 229.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:28:15.588 INFO:journalctl@ceph.mon.c.smithi029.stdout: Interval compaction: 0.00 GB write, 229.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:28:15.588 INFO:journalctl@ceph.mon.c.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:28:15.588 INFO:journalctl@ceph.mon.c.smithi029.stdout: Block cache BinnedLRUCache@0x55c64da0b350#2 capacity: 512.00 MB usage: 3.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-09-17T09:28:15.588 INFO:journalctl@ceph.mon.c.smithi029.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.66 KB,0.00012517%) IndexBlock(1,2.83 KB,0.000539422%) Misc(1,0.00 KB,0%) 2024-09-17T09:28:15.588 INFO:journalctl@ceph.mon.c.smithi029.stdout: 2024-09-17T09:28:15.589 INFO:journalctl@ceph.mon.c.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:28:15.589 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: starting mon.c rank 1 at public addrs [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] at bind addrs [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:28:15.589 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: mon.c@-1(???) e3 preinit fsid 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:28:15.589 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: mon.c@-1(???).mds e1 new map 2024-09-17T09:28:15.589 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: mon.c@-1(???).mds e1 print_map 2024-09-17T09:28:15.589 INFO:journalctl@ceph.mon.c.smithi029.stdout: e1 2024-09-17T09:28:15.589 INFO:journalctl@ceph.mon.c.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T09:28:15.589 INFO:journalctl@ceph.mon.c.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T09:28:15.589 INFO:journalctl@ceph.mon.c.smithi029.stdout: legacy client fscid: -1 2024-09-17T09:28:15.589 INFO:journalctl@ceph.mon.c.smithi029.stdout: 2024-09-17T09:28:15.590 INFO:journalctl@ceph.mon.c.smithi029.stdout: No filesystems configured 2024-09-17T09:28:15.590 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: mon.c@-1(???).osd e78 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-17T09:28:15.590 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: mon.c@-1(???).osd e78 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T09:28:15.590 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: mon.c@-1(???).osd e78 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T09:28:15.590 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: mon.c@-1(???).osd e78 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T09:28:15.590 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 ceph-mon[60489]: mon.c@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-09-17T09:28:15.590 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 bash[60470]: 0d3d1a65e2c90bcd7059d6fffa435afdbbb8f998f6048dd6f5c04723ffdc3154 2024-09-17T09:28:15.590 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:15 smithi029 systemd[1]: Started Ceph mon.c for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:16.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:12.530713+0000 mgr.y (mgr.25264) 21 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:28:16.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:13.375401+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:28:16.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:13.375623+0000 mon.b (mon.2) 91 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:28:16.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:13.375806+0000 mon.b (mon.2) 92 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:28:16.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:14.531074+0000 mgr.y (mgr.25264) 22 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:28:16.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:15.439701+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2024-09-17T09:28:16.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:15.524593+0000 mon.a (mon.0) 46 : cluster [INF] mon.a calling monitor election 2024-09-17T09:28:16.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:15.525879+0000 mon.a (mon.0) 47 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T09:28:16.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:15.530335+0000 mon.a (mon.0) 48 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:28:16.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:15.530854+0000 mon.a (mon.0) 49 : cluster [DBG] fsmap 2024-09-17T09:28:16.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:15.530875+0000 mon.a (mon.0) 50 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-17T09:28:16.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:15.531175+0000 mon.a (mon.0) 51 : cluster [DBG] mgrmap e33: y(active, since 16s), standbys: x 2024-09-17T09:28:16.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:15.531310+0000 mon.a (mon.0) 52 : cluster [WRN] Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:28:16.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:15.531324+0000 mon.a (mon.0) 53 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:16.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:15.531332+0000 mon.a (mon.0) 54 : cluster [WRN] daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:16.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:15.531340+0000 mon.a (mon.0) 55 : cluster [WRN] daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:16.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:15.537027+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:16.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:15.542777+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:18.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:16.531912+0000 mgr.y (mgr.25264) 23 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 91 B/s rd, 0 op/s 2024-09-17T09:28:18.349 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 4. 2024-09-17T09:28:18.349 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:18.349 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:28:18.349 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 podman[49383]: 2024-09-17 09:28:18.084328775 +0000 UTC m=+0.016733609 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:28:18.349 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 podman[49383]: 2024-09-17 09:28:18.233808573 +0000 UTC m=+0.166213456 container create 5ab5993835a2c5a8002020f053222f013ddf40654f0bf8f41409a553e2fe0cbd (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:18.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 podman[49383]: 2024-09-17 09:28:18.48594796 +0000 UTC m=+0.418352801 container init 5ab5993835a2c5a8002020f053222f013ddf40654f0bf8f41409a553e2fe0cbd (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:18.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 podman[49383]: 2024-09-17 09:28:18.488653699 +0000 UTC m=+0.421058521 container start 5ab5993835a2c5a8002020f053222f013ddf40654f0bf8f41409a553e2fe0cbd (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:18.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.518Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:28:18.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.518Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:28:18.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.518Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:28:18.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.518Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:28:18.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.518Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:28:18.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.518Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:28:18.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.519Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:28:18.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.519Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:28:18.757 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.522Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:28:18.757 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.522Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=11 2024-09-17T09:28:18.757 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.529Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=11 2024-09-17T09:28:18.757 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.529Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=11 2024-09-17T09:28:18.757 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.529Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=11 2024-09-17T09:28:18.757 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.529Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=11 2024-09-17T09:28:18.757 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.529Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=11 2024-09-17T09:28:18.757 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.529Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=11 2024-09-17T09:28:18.758 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.530Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=11 2024-09-17T09:28:18.758 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.530Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=11 2024-09-17T09:28:18.758 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.530Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=11 2024-09-17T09:28:18.758 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.530Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=11 2024-09-17T09:28:18.758 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.531Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=11 2024-09-17T09:28:18.758 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.531Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=8.910366ms 2024-09-17T09:28:18.758 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.534Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:28:18.758 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.534Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:28:18.758 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.534Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:28:18.758 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.534Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:28:18.758 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.534Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:28:18.758 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.534Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:28:18.759 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.534Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:28:18.759 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.534Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:28:18.759 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.534Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:28:18.759 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.534Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:28:18.759 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.534Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:28:18.759 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.534Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:28:18.759 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=info ts=2024-09-17T09:28:18.534Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:28:18.759 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[49398]: level=error ts=2024-09-17T09:28:18.534Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:28:18.759 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 bash[49383]: 5ab5993835a2c5a8002020f053222f013ddf40654f0bf8f41409a553e2fe0cbd 2024-09-17T09:28:18.759 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:18.759 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:18 smithi103 podman[49415]: 2024-09-17 09:28:18.593017738 +0000 UTC m=+0.050989723 container died 5ab5993835a2c5a8002020f053222f013ddf40654f0bf8f41409a553e2fe0cbd (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:19.505 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:19 smithi103 podman[49415]: 2024-09-17 09:28:19.148921832 +0000 UTC m=+0.606893825 container remove 5ab5993835a2c5a8002020f053222f013ddf40654f0bf8f41409a553e2fe0cbd (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:19.505 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:19 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:28:19.506 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:19 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:28:20.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:19 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:18.532526+0000 mgr.y (mgr.25264) 24 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 91 B/s rd, 0 op/s 2024-09-17T09:28:20.548 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:20 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:28:20.237+0000 7fe73b7e7640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-17T09:28:21.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:20.117728+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:21.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:20.123996+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:21.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:20.645056+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:21.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:20.652000+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:22.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:20.533211+0000 mgr.y (mgr.25264) 25 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:24.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:24 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:22.533695+0000 mgr.y (mgr.25264) 26 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:25.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:24.223022+0000 mon.a (mon.0) 62 : audit [DBG] from='client.44232 172.21.15.29:0/840260197' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:25.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:24.524362+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:25.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:24.533533+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:25.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:24.536435+0000 mon.b (mon.2) 93 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:25.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:24.537240+0000 mon.b (mon.2) 94 : audit [INF] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:28:25.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:24.542528+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:25.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:24.564685+0000 mon.a (mon.0) 66 : audit [INF] from='client.54121 172.21.15.29:0/3440964050' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:28:25.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:24.665402+0000 mon.b (mon.2) 95 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:28:25.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:24.669700+0000 mon.b (mon.2) 96 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:25.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:24.671125+0000 mon.b (mon.2) 97 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T09:28:25.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:24.672068+0000 mon.b (mon.2) 98 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T09:28:26.443 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:24.534025+0000 mgr.y (mgr.25264) 27 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:26.443 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:28:24.537964+0000 mgr.y (mgr.25264) 28 : cephadm [WRN] unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:26.443 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:24.538414+0000 mgr.y (mgr.25264) 29 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:26.443 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cluster 2024-09-17T09:28:24.538565+0000 mgr.y (mgr.25264) 30 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:26.443 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:28:24.672611+0000 mgr.y (mgr.25264) 31 : cephadm [INF] Upgrade: It appears safe to stop mon.b 2024-09-17T09:28:26.443 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:26.121455+0000 mon.a (mon.0) 67 : audit [INF] from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:26.443 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:26.122467+0000 mon.b (mon.2) 99 : audit [INF] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:26.443 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:26 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:26.123369+0000 mon.a (mon.0) 68 : audit [DBG] from='mgr.25264 ' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:27.417 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:28:26.116024+0000 mgr.y (mgr.25264) 32 : cephadm [INF] Upgrade: Updating mon.b 2024-09-17T09:28:27.418 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: audit 2024-09-17T09:28:26.124272+0000 mon.b (mon.2) 100 : audit [DBG] from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:27.418 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: cephadm 2024-09-17T09:28:26.128616+0000 mgr.y (mgr.25264) 33 : cephadm [INF] Deploying daemon mon.b on smithi103 2024-09-17T09:28:28.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:27 smithi103 systemd[1]: Stopping Ceph mon.b for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:28:28.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:28:27.871+0000 7fd2b5530700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-17T09:28:28.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:27 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b[23885]: debug 2024-09-17T09:28:27.871+0000 7fd2b5530700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-17T09:28:28.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:27 smithi103 podman[49758]: 2024-09-17 09:28:27.893161651 +0000 UTC m=+0.137071439 container died 203d1f26d7b87eb313b794f6ec7b722b3a1300f8097a11f1025a2662ca17f10c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-09-17T09:28:28.451 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:28 smithi103 podman[49758]: 2024-09-17 09:28:28.281737518 +0000 UTC m=+0.525647337 container cleanup 203d1f26d7b87eb313b794f6ec7b722b3a1300f8097a11f1025a2662ca17f10c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b, CEPH_POINT_RELEASE=-16.2.0, maintainer=Dimitri Savineau , RELEASE=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.schema-version=1.0) 2024-09-17T09:28:28.451 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:28 smithi103 bash[49758]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b 2024-09-17T09:28:28.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:28 smithi103 podman[49770]: 2024-09-17 09:28:28.450345033 +0000 UTC m=+0.559945269 container remove 203d1f26d7b87eb313b794f6ec7b722b3a1300f8097a11f1025a2662ca17f10c (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon.b, RELEASE=HEAD, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-17T09:28:29.221 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:28 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.b.service: Deactivated successfully. 2024-09-17T09:28:29.221 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:28 smithi103 systemd[1]: Stopped Ceph mon.b for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:29.221 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 systemd[1]: Starting Ceph mon.b for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:28:29.479 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 podman[49914]: 2024-09-17 09:28:29.12329518 +0000 UTC m=+0.016248973 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:28:29.479 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 podman[49914]: 2024-09-17 09:28:29.243084866 +0000 UTC m=+0.136038659 container create da050285f124fc567176c596942e5fef7d87666a4bd27b293c193dc80cd9accd (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-b, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:28:29.479 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:29 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 5. 2024-09-17T09:28:29.479 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:29 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:29.480 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:29 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:28:29.755 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:29 smithi103 podman[49955]: 2024-09-17 09:28:29.590408087 +0000 UTC m=+0.196456567 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:28:29.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:29 smithi103 podman[49955]: 2024-09-17 09:28:29.712806771 +0000 UTC m=+0.318855295 container create cf0e8018096e867726985892e0b758cfa89b0ca864ba18b9a682dd2e8b67e6d8 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:29.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 podman[49914]: 2024-09-17 09:28:29.478331011 +0000 UTC m=+0.371284804 container init da050285f124fc567176c596942e5fef7d87666a4bd27b293c193dc80cd9accd (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=reef-19e2a2b) 2024-09-17T09:28:29.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 podman[49914]: 2024-09-17 09:28:29.482162978 +0000 UTC m=+0.375116772 container start da050285f124fc567176c596942e5fef7d87666a4bd27b293c193dc80cd9accd (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-b, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:28:29.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T09:28:29.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable), process ceph-mon, pid 2 2024-09-17T09:28:29.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: pidfile_write: ignore empty --pid-file 2024-09-17T09:28:29.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: load: jerasure load: lrc 2024-09-17T09:28:29.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: RocksDB version: 7.9.2 2024-09-17T09:28:29.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Git sha 0 2024-09-17T09:28:29.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Compile date 2024-09-16 15:15:13 2024-09-17T09:28:29.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: DB SUMMARY 2024-09-17T09:28:29.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: DB Session ID: K0V4SQVB115DCLL1TYHP 2024-09-17T09:28:29.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: CURRENT file: CURRENT 2024-09-17T09:28:29.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T09:28:29.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: MANIFEST file: MANIFEST-000005 size: 896 Bytes 2024-09-17T09:28:29.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000020.sst 2024-09-17T09:28:29.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000018.log size: 7599865 ; 2024-09-17T09:28:29.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.error_if_exists: 0 2024-09-17T09:28:29.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.create_if_missing: 0 2024-09-17T09:28:29.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.paranoid_checks: 1 2024-09-17T09:28:29.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T09:28:29.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T09:28:29.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T09:28:29.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.env: 0x555df2a6ebc0 2024-09-17T09:28:29.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T09:28:29.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.info_log: 0x555df3590be0 2024-09-17T09:28:29.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T09:28:29.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.statistics: (nil) 2024-09-17T09:28:29.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.use_fsync: 0 2024-09-17T09:28:29.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_log_file_size: 0 2024-09-17T09:28:29.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T09:28:29.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T09:28:29.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T09:28:29.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T09:28:29.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.allow_fallocate: 1 2024-09-17T09:28:29.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T09:28:29.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T09:28:29.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.use_direct_reads: 0 2024-09-17T09:28:29.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T09:28:29.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T09:28:29.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.db_log_dir: 2024-09-17T09:28:29.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.wal_dir: 2024-09-17T09:28:29.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T09:28:29.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T09:28:29.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T09:28:29.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T09:28:29.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T09:28:29.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T09:28:29.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T09:28:29.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T09:28:29.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.write_buffer_manager: 0x555df359eb40 2024-09-17T09:28:29.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T09:28:29.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T09:28:29.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T09:28:29.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T09:28:29.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T09:28:29.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T09:28:29.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T09:28:29.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T09:28:29.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.unordered_write: 0 2024-09-17T09:28:29.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T09:28:29.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T09:28:29.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T09:28:29.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T09:28:29.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.row_cache: None 2024-09-17T09:28:29.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.wal_filter: None 2024-09-17T09:28:29.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T09:28:29.763 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T09:28:29.763 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.two_write_queues: 0 2024-09-17T09:28:29.763 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T09:28:29.763 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.wal_compression: 0 2024-09-17T09:28:29.763 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.atomic_flush: 0 2024-09-17T09:28:29.764 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T09:28:29.764 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T09:28:29.764 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T09:28:29.764 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.log_readahead_size: 0 2024-09-17T09:28:29.764 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T09:28:29.764 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T09:28:29.764 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T09:28:29.764 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T09:28:29.764 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T09:28:29.764 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T09:28:29.764 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T09:28:29.764 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_background_jobs: 2 2024-09-17T09:28:29.764 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_background_compactions: -1 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_subcompactions: 1 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_open_files: -1 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T09:28:29.765 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T09:28:29.766 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_background_flushes: -1 2024-09-17T09:28:29.766 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Compression algorithms supported: 2024-09-17T09:28:29.766 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: kZSTD supported: 0 2024-09-17T09:28:29.766 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: kXpressCompression supported: 0 2024-09-17T09:28:29.766 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: kBZip2Compression supported: 0 2024-09-17T09:28:29.766 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T09:28:29.766 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: kLZ4Compression supported: 1 2024-09-17T09:28:29.766 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: kZlibCompression supported: 1 2024-09-17T09:28:29.766 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T09:28:29.766 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: kSnappyCompression supported: 1 2024-09-17T09:28:29.766 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T09:28:29.766 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T09:28:29.766 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-17T09:28:29.767 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T09:28:29.767 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T09:28:29.767 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.merge_operator: 2024-09-17T09:28:29.767 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compaction_filter: None 2024-09-17T09:28:29.767 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T09:28:29.767 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T09:28:29.767 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T09:28:29.767 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T09:28:29.767 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555df35907e0) 2024-09-17T09:28:29.767 INFO:journalctl@ceph.mon.b.smithi103.stdout: cache_index_and_filter_blocks: 1 2024-09-17T09:28:29.767 INFO:journalctl@ceph.mon.b.smithi103.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T09:28:29.767 INFO:journalctl@ceph.mon.b.smithi103.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T09:28:29.767 INFO:journalctl@ceph.mon.b.smithi103.stdout: pin_top_level_index_and_filter: 1 2024-09-17T09:28:29.768 INFO:journalctl@ceph.mon.b.smithi103.stdout: index_type: 0 2024-09-17T09:28:29.768 INFO:journalctl@ceph.mon.b.smithi103.stdout: data_block_index_type: 0 2024-09-17T09:28:29.768 INFO:journalctl@ceph.mon.b.smithi103.stdout: index_shortening: 1 2024-09-17T09:28:29.768 INFO:journalctl@ceph.mon.b.smithi103.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T09:28:29.768 INFO:journalctl@ceph.mon.b.smithi103.stdout: checksum: 4 2024-09-17T09:28:29.768 INFO:journalctl@ceph.mon.b.smithi103.stdout: no_block_cache: 0 2024-09-17T09:28:29.768 INFO:journalctl@ceph.mon.b.smithi103.stdout: block_cache: 0x555df3585350 2024-09-17T09:28:29.768 INFO:journalctl@ceph.mon.b.smithi103.stdout: block_cache_name: BinnedLRUCache 2024-09-17T09:28:29.768 INFO:journalctl@ceph.mon.b.smithi103.stdout: block_cache_options: 2024-09-17T09:28:29.768 INFO:journalctl@ceph.mon.b.smithi103.stdout: capacity : 536870912 2024-09-17T09:28:29.768 INFO:journalctl@ceph.mon.b.smithi103.stdout: num_shard_bits : 4 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: strict_capacity_limit : 0 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: high_pri_pool_ratio: 0.000 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: block_cache_compressed: (nil) 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: persistent_cache: (nil) 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: block_size: 4096 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: block_size_deviation: 10 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: block_restart_interval: 16 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: index_block_restart_interval: 1 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: metadata_block_size: 4096 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: partition_filters: 0 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: use_delta_encoding: 1 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: filter_policy: bloomfilter 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: whole_key_filtering: 1 2024-09-17T09:28:29.769 INFO:journalctl@ceph.mon.b.smithi103.stdout: verify_compression: 0 2024-09-17T09:28:29.770 INFO:journalctl@ceph.mon.b.smithi103.stdout: read_amp_bytes_per_bit: 0 2024-09-17T09:28:29.770 INFO:journalctl@ceph.mon.b.smithi103.stdout: format_version: 5 2024-09-17T09:28:29.770 INFO:journalctl@ceph.mon.b.smithi103.stdout: enable_index_compression: 1 2024-09-17T09:28:29.770 INFO:journalctl@ceph.mon.b.smithi103.stdout: block_align: 0 2024-09-17T09:28:29.770 INFO:journalctl@ceph.mon.b.smithi103.stdout: max_auto_readahead_size: 262144 2024-09-17T09:28:29.770 INFO:journalctl@ceph.mon.b.smithi103.stdout: prepopulate_block_cache: 0 2024-09-17T09:28:29.770 INFO:journalctl@ceph.mon.b.smithi103.stdout: initial_auto_readahead_size: 8192 2024-09-17T09:28:29.770 INFO:journalctl@ceph.mon.b.smithi103.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T09:28:29.770 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T09:28:29.770 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T09:28:29.770 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compression: NoCompression 2024-09-17T09:28:29.770 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T09:28:29.770 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.num_levels: 7 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T09:28:29.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T09:28:29.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T09:28:29.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T09:28:29.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T09:28:29.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T09:28:29.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T09:28:29.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T09:28:29.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T09:28:29.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T09:28:29.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T09:28:29.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T09:28:29.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T09:28:29.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T09:28:29.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T09:28:29.773 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T09:28:29.774 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.inplace_update_support: 0 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.bloom_locality: 0 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.max_successive_merges: 0 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.ttl: 2592000 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T09:28:29.775 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T09:28:29.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T09:28:29.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.enable_blob_files: false 2024-09-17T09:28:29.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.min_blob_size: 0 2024-09-17T09:28:29.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T09:28:29.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T09:28:29.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T09:28:29.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T09:28:29.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T09:28:29.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T09:28:29.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T09:28:29.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T09:28:29.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 20.sst 2024-09-17T09:28:29.776 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T09:28:29.777 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 22, last_sequence is 12018, log_number is 18,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T09:28:29.777 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 18 2024-09-17T09:28:29.777 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7793db32-aecd-41bf-8599-1834d4db54b6 2024-09-17T09:28:29.777 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726565309508099, "job": 1, "event": "recovery_started", "wal_files": [18]} 2024-09-17T09:28:29.777 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #18 mode 2 2024-09-17T09:28:29.777 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726565309525016, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 23, "file_size": 4883412, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12015, "largest_seqno": 12905, "table_properties": {"data_size": 4877737, "index_size": 3862, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 6020, "raw_average_key_size": 21, "raw_value_size": 4869419, "raw_average_value_size": 17085, "num_data_blocks": 172, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726565309, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7793db32-aecd-41bf-8599-1834d4db54b6", "db_session_id": "K0V4SQVB115DCLL1TYHP", "orig_file_number": 23, "seqno_to_time_mapping": "N/A"}} 2024-09-17T09:28:29.777 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726565309525140, "job": 1, "event": "recovery_finished"} 2024-09-17T09:28:29.777 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [db/version_set.cc:5047] Creating manifest 25 2024-09-17T09:28:29.777 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T09:28:29.777 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000018.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T09:28:29.777 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555df35b0e00 2024-09-17T09:28:29.777 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: DB pointer 0x555df36a4000 2024-09-17T09:28:29.777 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: ** DB Stats ** 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: ** Compaction Stats [default] ** 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: L0 1/0 4.66 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 581.3 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-17T09:28:29.778 INFO:journalctl@ceph.mon.b.smithi103.stdout: L6 1/0 24.60 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-17T09:28:29.779 INFO:journalctl@ceph.mon.b.smithi103.stdout: Sum 2/0 29.26 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 581.3 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-17T09:28:29.779 INFO:journalctl@ceph.mon.b.smithi103.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 581.3 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-17T09:28:29.779 INFO:journalctl@ceph.mon.b.smithi103.stdout: 2024-09-17T09:28:29.779 INFO:journalctl@ceph.mon.b.smithi103.stdout: ** Compaction Stats [default] ** 2024-09-17T09:28:29.779 INFO:journalctl@ceph.mon.b.smithi103.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T09:28:29.779 INFO:journalctl@ceph.mon.b.smithi103.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T09:28:29.779 INFO:journalctl@ceph.mon.b.smithi103.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 581.3 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-17T09:28:29.779 INFO:journalctl@ceph.mon.b.smithi103.stdout: 2024-09-17T09:28:29.779 INFO:journalctl@ceph.mon.b.smithi103.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T09:28:29.779 INFO:journalctl@ceph.mon.b.smithi103.stdout: 2024-09-17T09:28:29.779 INFO:journalctl@ceph.mon.b.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T09:28:29.779 INFO:journalctl@ceph.mon.b.smithi103.stdout: Flush(GB): cumulative 0.005, interval 0.005 2024-09-17T09:28:29.779 INFO:journalctl@ceph.mon.b.smithi103.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T09:28:29.780 INFO:journalctl@ceph.mon.b.smithi103.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T09:28:29.780 INFO:journalctl@ceph.mon.b.smithi103.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T09:28:29.780 INFO:journalctl@ceph.mon.b.smithi103.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T09:28:29.780 INFO:journalctl@ceph.mon.b.smithi103.stdout: Cumulative compaction: 0.00 GB write, 235.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:28:29.780 INFO:journalctl@ceph.mon.b.smithi103.stdout: Interval compaction: 0.00 GB write, 235.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T09:28:29.780 INFO:journalctl@ceph.mon.b.smithi103.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T09:28:29.780 INFO:journalctl@ceph.mon.b.smithi103.stdout: Block cache BinnedLRUCache@0x555df3585350#2 capacity: 512.00 MB usage: 4.98 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-17T09:28:29.780 INFO:journalctl@ceph.mon.b.smithi103.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.91 KB,0.000172853%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-09-17T09:28:29.780 INFO:journalctl@ceph.mon.b.smithi103.stdout: 2024-09-17T09:28:29.780 INFO:journalctl@ceph.mon.b.smithi103.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T09:28:29.780 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: starting mon.b rank 2 at public addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] at bind addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:28:29.780 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: mon.b@-1(???) e3 preinit fsid 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:28:29.780 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: mon.b@-1(???).mds e1 new map 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: mon.b@-1(???).mds e1 print_map 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout: e1 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout: legacy client fscid: -1 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout: 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout: No filesystems configured 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: mon.b@-1(???).osd e78 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: mon.b@-1(???).osd e78 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: mon.b@-1(???).osd e78 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: mon.b@-1(???).osd e78 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 ceph-mon[49967]: mon.b@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 bash[49914]: da050285f124fc567176c596942e5fef7d87666a4bd27b293c193dc80cd9accd 2024-09-17T09:28:29.781 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:29 smithi103 systemd[1]: Started Ceph mon.b for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:30.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:29 smithi103 podman[49955]: 2024-09-17 09:28:29.990710096 +0000 UTC m=+0.596758582 container init cf0e8018096e867726985892e0b758cfa89b0ca864ba18b9a682dd2e8b67e6d8 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:30.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:29 smithi103 podman[49955]: 2024-09-17 09:28:29.993579113 +0000 UTC m=+0.599627588 container start cf0e8018096e867726985892e0b758cfa89b0ca864ba18b9a682dd2e8b67e6d8 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:30.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.023Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:28:30.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.023Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:28:30.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.023Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:28:30.257 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.023Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:28:30.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.023Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:28:30.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.023Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:28:30.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.025Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:28:30.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.025Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:28:30.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.027Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:28:30.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.027Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=12 2024-09-17T09:28:30.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.034Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=12 2024-09-17T09:28:30.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.034Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=12 2024-09-17T09:28:30.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.034Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=12 2024-09-17T09:28:30.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.034Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=12 2024-09-17T09:28:30.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.035Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=12 2024-09-17T09:28:30.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.035Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=12 2024-09-17T09:28:30.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.035Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=12 2024-09-17T09:28:30.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.035Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=12 2024-09-17T09:28:30.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.035Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=12 2024-09-17T09:28:30.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.035Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=12 2024-09-17T09:28:30.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.035Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=12 2024-09-17T09:28:30.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.037Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=12 maxSegment=12 2024-09-17T09:28:30.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.037Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=9.816286ms 2024-09-17T09:28:30.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.039Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:28:30.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.039Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:28:30.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.039Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:28:30.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.039Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:28:30.262 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.039Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:28:30.262 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.039Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:28:30.262 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.039Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:28:30.262 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.039Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:28:30.263 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.039Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:28:30.263 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.039Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:28:30.263 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.039Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:28:30.263 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.039Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:28:30.263 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=info ts=2024-09-17T09:28:30.039Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:28:30.264 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[50013]: level=error ts=2024-09-17T09:28:30.039Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:28:30.264 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 bash[49955]: cf0e8018096e867726985892e0b758cfa89b0ca864ba18b9a682dd2e8b67e6d8 2024-09-17T09:28:30.264 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:30.264 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 podman[50030]: 2024-09-17 09:28:30.117753166 +0000 UTC m=+0.070729418 container died cf0e8018096e867726985892e0b758cfa89b0ca864ba18b9a682dd2e8b67e6d8 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:31.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 podman[50030]: 2024-09-17 09:28:30.67911419 +0000 UTC m=+0.632090430 container remove cf0e8018096e867726985892e0b758cfa89b0ca864ba18b9a682dd2e8b67e6d8 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:31.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:28:31.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:30 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:28:37.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mkfs 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:28:37.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T09:28:37.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T09:28:37.509 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring daemon node-exporter.a on smithi029 2024-09-17T09:28:37.509 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T09:28:37.509 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring daemon alertmanager.a on smithi029 2024-09-17T09:28:37.509 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring iscsi.foo.smithi029.gellkh (dependencies changed)... 2024-09-17T09:28:37.509 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:28:37.510 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:28:37.510 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:28:37.510 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T09:28:37.510 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring daemon node-exporter.b on smithi103 2024-09-17T09:28:37.510 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring grafana.a (dependencies changed)... 2024-09-17T09:28:37.510 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T09:28:37.511 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring daemon grafana.a on smithi103 2024-09-17T09:28:37.511 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T09:28:37.511 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T09:28:37.511 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Upgrade: Updating mgr.y 2024-09-17T09:28:37.511 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Deploying daemon mgr.y on smithi029 2024-09-17T09:28:37.512 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.512 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: inventory: adjusted host smithi103 addr 'smithi103' -> '172.21.15.103' 2024-09-17T09:28:37.512 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [17/Sep/2024:09:27:15] ENGINE Bus STARTING 2024-09-17T09:28:37.512 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [17/Sep/2024:09:27:15] ENGINE Serving on http://172.21.15.29:8765 2024-09-17T09:28:37.512 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [17/Sep/2024:09:27:16] ENGINE Serving on https://172.21.15.29:7150 2024-09-17T09:28:37.512 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [17/Sep/2024:09:27:16] ENGINE Bus STARTED 2024-09-17T09:28:37.513 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [17/Sep/2024:09:27:16] ENGINE Error in HTTPServer.serve 2024-09-17T09:28:37.513 INFO:journalctl@ceph.mon.b.smithi103.stdout: Traceback (most recent call last): 2024-09-17T09:28:37.513 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:28:37.513 INFO:journalctl@ceph.mon.b.smithi103.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:28:37.513 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:28:37.513 INFO:journalctl@ceph.mon.b.smithi103.stdout: self._run(expiration_interval) 2024-09-17T09:28:37.514 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:28:37.514 INFO:journalctl@ceph.mon.b.smithi103.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:28:37.514 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:28:37.514 INFO:journalctl@ceph.mon.b.smithi103.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:28:37.514 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:28:37.515 INFO:journalctl@ceph.mon.b.smithi103.stdout: s = self.context.wrap_socket( 2024-09-17T09:28:37.515 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:28:37.515 INFO:journalctl@ceph.mon.b.smithi103.stdout: return self.sslsocket_class._create( 2024-09-17T09:28:37.515 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:28:37.515 INFO:journalctl@ceph.mon.b.smithi103.stdout: self.do_handshake() 2024-09-17T09:28:37.515 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:28:37.516 INFO:journalctl@ceph.mon.b.smithi103.stdout: self._sslobj.do_handshake() 2024-09-17T09:28:37.516 INFO:journalctl@ceph.mon.b.smithi103.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:28:37.516 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.516 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring iscsi.foo.smithi029.gellkh (dependencies changed)... 2024-09-17T09:28:37.516 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:28:37.517 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:28:37.517 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:28:37.517 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Upgrade: Updating mgr.x 2024-09-17T09:28:37.517 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Deploying daemon mgr.x on smithi103 2024-09-17T09:28:37.517 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.517 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Upgrade: Setting container_image for all mgr 2024-09-17T09:28:37.517 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Upgrade: It appears safe to stop mon.a 2024-09-17T09:28:37.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Upgrade: Updating mon.a 2024-09-17T09:28:37.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Deploying daemon mon.a on smithi029 2024-09-17T09:28:37.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: fsmap 2024-09-17T09:28:37.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: osdmap e77: 8 total, 8 up, 8 in 2024-09-17T09:28:37.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mgrmap e26: y(active, since 41s), standbys: x 2024-09-17T09:28:37.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:28:37.519 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:37.519 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:37.519 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:37.519 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mgrmap e27: y(active, since 41s), standbys: x 2024-09-17T09:28:37.519 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mgrmap e28: y(active, since 42s), standbys: x 2024-09-17T09:28:37.519 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Standby manager daemon x restarted 2024-09-17T09:28:37.519 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Standby manager daemon x started 2024-09-17T09:28:37.520 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mgrmap e29: y(active, since 45s), standbys: x 2024-09-17T09:28:37.520 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Active manager daemon y restarted 2024-09-17T09:28:37.520 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Activating manager daemon y 2024-09-17T09:28:37.520 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T09:28:37.520 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mgrmap e30: y(active, starting, since 0.00714194s), standbys: x 2024-09-17T09:28:37.520 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Manager daemon y is now available 2024-09-17T09:28:37.520 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mgrmap e31: y(active, since 1.01065s), standbys: x 2024-09-17T09:28:37.521 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.521 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T09:28:37.522 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Cluster is now healthy 2024-09-17T09:28:37.522 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mgrmap e32: y(active, since 2s), standbys: x 2024-09-17T09:28:37.522 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [17/Sep/2024:09:28:02] ENGINE Bus STARTING 2024-09-17T09:28:37.522 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [17/Sep/2024:09:28:02] ENGINE Serving on http://172.21.15.29:8765 2024-09-17T09:28:37.522 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [17/Sep/2024:09:28:02] ENGINE Serving on https://172.21.15.29:7150 2024-09-17T09:28:37.522 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [17/Sep/2024:09:28:02] ENGINE Bus STARTED 2024-09-17T09:28:37.523 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [17/Sep/2024:09:28:02] ENGINE Error in HTTPServer.serve 2024-09-17T09:28:37.523 INFO:journalctl@ceph.mon.b.smithi103.stdout: Traceback (most recent call last): 2024-09-17T09:28:37.523 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:28:37.523 INFO:journalctl@ceph.mon.b.smithi103.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:28:37.523 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:28:37.524 INFO:journalctl@ceph.mon.b.smithi103.stdout: self._run(expiration_interval) 2024-09-17T09:28:37.524 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:28:37.524 INFO:journalctl@ceph.mon.b.smithi103.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:28:37.524 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:28:37.524 INFO:journalctl@ceph.mon.b.smithi103.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:28:37.524 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:28:37.524 INFO:journalctl@ceph.mon.b.smithi103.stdout: s = self.context.wrap_socket( 2024-09-17T09:28:37.524 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:28:37.524 INFO:journalctl@ceph.mon.b.smithi103.stdout: return self.sslsocket_class._create( 2024-09-17T09:28:37.524 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:28:37.525 INFO:journalctl@ceph.mon.b.smithi103.stdout: self.do_handshake() 2024-09-17T09:28:37.525 INFO:journalctl@ceph.mon.b.smithi103.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:28:37.525 INFO:journalctl@ceph.mon.b.smithi103.stdout: self._sslobj.do_handshake() 2024-09-17T09:28:37.525 INFO:journalctl@ceph.mon.b.smithi103.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:28:37.525 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.525 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mgrmap e33: y(active, since 4s), standbys: x 2024-09-17T09:28:37.525 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.526 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.526 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-17T09:28:37.526 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:37.526 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:37.526 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:37.526 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/1781687977' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:37.526 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:37.526 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T09:28:37.527 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='client.44190 172.21.15.29:0/197166824' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:28:37.527 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.527 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.527 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:37.528 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:28:37.528 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.528 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-17T09:28:37.528 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.528 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:28:37.528 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:37.528 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T09:28:37.528 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T09:28:37.528 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Upgrade: It appears safe to stop mon.c 2024-09-17T09:28:37.529 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/871367148' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:37.529 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:28:37.529 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='client.44214 172.21.15.29:0/552312000' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]: access denied 2024-09-17T09:28:37.529 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Upgrade: Updating mon.c 2024-09-17T09:28:37.529 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.529 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:37.529 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:37.530 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:37.530 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Deploying daemon mon.c on smithi029 2024-09-17T09:28:37.530 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:28:37.530 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:28:37.530 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:28:37.530 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:28:37.530 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:28:37.530 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:28:37.530 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mon.c calling monitor election 2024-09-17T09:28:37.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mon.a calling monitor election 2024-09-17T09:28:37.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T09:28:37.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:28:37.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: fsmap 2024-09-17T09:28:37.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T09:28:37.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mgrmap e33: y(active, since 16s), standbys: x 2024-09-17T09:28:37.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:28:37.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:37.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:37.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:37.532 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.532 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.532 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 91 B/s rd, 0 op/s 2024-09-17T09:28:37.532 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 91 B/s rd, 0 op/s 2024-09-17T09:28:37.532 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.532 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.532 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:37.533 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.533 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.533 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.533 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='client.44232 172.21.15.29:0/840260197' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:37.533 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.533 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.533 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:28:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='client.54121 172.21.15.29:0/3440964050' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:28:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:28:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T09:28:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T09:28:37.534 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Upgrade: It appears safe to stop mon.b 2024-09-17T09:28:37.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Upgrade: Updating mon.b 2024-09-17T09:28:37.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:37.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 ' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:37.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:37.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Deploying daemon mon.b on smithi103 2024-09-17T09:28:37.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T09:28:37.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:28:37.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:28:37.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:28:37.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mon.b calling monitor election 2024-09-17T09:28:37.536 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mon.c calling monitor election 2024-09-17T09:28:37.536 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); 1/3 mons down, quorum a,c 2024-09-17T09:28:37.536 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:37.536 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:37.536 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:37.536 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-17T09:28:37.536 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mon.b (rank 2) addr [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] is down (out of quorum) 2024-09-17T09:28:37.536 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:28:37.536 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:28:37.536 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:28:37.536 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mon.a calling monitor election 2024-09-17T09:28:37.537 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T09:28:37.537 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: monmap e4: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:28:37.537 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: fsmap 2024-09-17T09:28:37.537 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T09:28:37.537 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: mgrmap e33: y(active, since 36s), standbys: x 2024-09-17T09:28:37.537 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-17T09:28:37.537 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:28:37.537 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:37.537 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:37.537 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:37 smithi103 ceph-mon[49967]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:37.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mkfs 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:28:37.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T09:28:37.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T09:28:37.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring daemon node-exporter.a on smithi029 2024-09-17T09:28:37.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T09:28:37.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring daemon alertmanager.a on smithi029 2024-09-17T09:28:37.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring iscsi.foo.smithi029.gellkh (dependencies changed)... 2024-09-17T09:28:37.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:28:37.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:28:37.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring daemon node-exporter.b on smithi103 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring grafana.a (dependencies changed)... 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring daemon grafana.a on smithi103 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Upgrade: Updating mgr.y 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Deploying daemon mgr.y on smithi029 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: inventory: adjusted host smithi103 addr 'smithi103' -> '172.21.15.103' 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [17/Sep/2024:09:27:15] ENGINE Bus STARTING 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [17/Sep/2024:09:27:15] ENGINE Serving on http://172.21.15.29:8765 2024-09-17T09:28:37.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [17/Sep/2024:09:27:16] ENGINE Serving on https://172.21.15.29:7150 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [17/Sep/2024:09:27:16] ENGINE Bus STARTED 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [17/Sep/2024:09:27:16] ENGINE Error in HTTPServer.serve 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout: Traceback (most recent call last): 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._run(expiration_interval) 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout: s = self.context.wrap_socket( 2024-09-17T09:28:37.551 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:28:37.552 INFO:journalctl@ceph.mon.a.smithi029.stdout: return self.sslsocket_class._create( 2024-09-17T09:28:37.552 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:28:37.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mkfs 69135c5a-74d5-11ef-bceb-c7b262605968 2024-09-17T09:28:37.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T09:28:37.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T09:28:37.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring daemon node-exporter.a on smithi029 2024-09-17T09:28:37.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T09:28:37.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring daemon alertmanager.a on smithi029 2024-09-17T09:28:37.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring iscsi.foo.smithi029.gellkh (dependencies changed)... 2024-09-17T09:28:37.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:28:37.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:28:37.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:28:37.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T09:28:37.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring daemon node-exporter.b on smithi103 2024-09-17T09:28:37.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring grafana.a (dependencies changed)... 2024-09-17T09:28:37.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T09:28:37.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring daemon grafana.a on smithi103 2024-09-17T09:28:37.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T09:28:37.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T09:28:37.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Upgrade: Updating mgr.y 2024-09-17T09:28:37.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Deploying daemon mgr.y on smithi029 2024-09-17T09:28:37.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: inventory: adjusted host smithi103 addr 'smithi103' -> '172.21.15.103' 2024-09-17T09:28:37.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [17/Sep/2024:09:27:15] ENGINE Bus STARTING 2024-09-17T09:28:37.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [17/Sep/2024:09:27:15] ENGINE Serving on http://172.21.15.29:8765 2024-09-17T09:28:37.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [17/Sep/2024:09:27:16] ENGINE Serving on https://172.21.15.29:7150 2024-09-17T09:28:37.554 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [17/Sep/2024:09:27:16] ENGINE Bus STARTED 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [17/Sep/2024:09:27:16] ENGINE Error in HTTPServer.serve 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout: Traceback (most recent call last): 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout: self._run(expiration_interval) 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout: s = self.context.wrap_socket( 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:28:37.555 INFO:journalctl@ceph.mon.c.smithi029.stdout: return self.sslsocket_class._create( 2024-09-17T09:28:37.556 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:28:37.556 INFO:journalctl@ceph.mon.a.smithi029.stdout: self.do_handshake() 2024-09-17T09:28:37.556 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:28:37.556 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._sslobj.do_handshake() 2024-09-17T09:28:37.556 INFO:journalctl@ceph.mon.a.smithi029.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:28:37.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring iscsi.foo.smithi029.gellkh (dependencies changed)... 2024-09-17T09:28:37.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:28:37.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:28:37.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:28:37.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Upgrade: Updating mgr.x 2024-09-17T09:28:37.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Deploying daemon mgr.x on smithi103 2024-09-17T09:28:37.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Upgrade: Setting container_image for all mgr 2024-09-17T09:28:37.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Upgrade: It appears safe to stop mon.a 2024-09-17T09:28:37.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Upgrade: Updating mon.a 2024-09-17T09:28:37.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Deploying daemon mon.a on smithi029 2024-09-17T09:28:37.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: fsmap 2024-09-17T09:28:37.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: osdmap e77: 8 total, 8 up, 8 in 2024-09-17T09:28:37.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mgrmap e26: y(active, since 41s), standbys: x 2024-09-17T09:28:37.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:28:37.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:37.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:37.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:37.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mgrmap e27: y(active, since 41s), standbys: x 2024-09-17T09:28:37.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mgrmap e28: y(active, since 42s), standbys: x 2024-09-17T09:28:37.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Standby manager daemon x restarted 2024-09-17T09:28:37.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Standby manager daemon x started 2024-09-17T09:28:37.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mgrmap e29: y(active, since 45s), standbys: x 2024-09-17T09:28:37.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Active manager daemon y restarted 2024-09-17T09:28:37.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Activating manager daemon y 2024-09-17T09:28:37.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T09:28:37.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mgrmap e30: y(active, starting, since 0.00714194s), standbys: x 2024-09-17T09:28:37.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Manager daemon y is now available 2024-09-17T09:28:37.558 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mgrmap e31: y(active, since 1.01065s), standbys: x 2024-09-17T09:28:37.559 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.559 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T09:28:37.559 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Cluster is now healthy 2024-09-17T09:28:37.559 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mgrmap e32: y(active, since 2s), standbys: x 2024-09-17T09:28:37.559 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [17/Sep/2024:09:28:02] ENGINE Bus STARTING 2024-09-17T09:28:37.559 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [17/Sep/2024:09:28:02] ENGINE Serving on http://172.21.15.29:8765 2024-09-17T09:28:37.559 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [17/Sep/2024:09:28:02] ENGINE Serving on https://172.21.15.29:7150 2024-09-17T09:28:37.559 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [17/Sep/2024:09:28:02] ENGINE Bus STARTED 2024-09-17T09:28:37.559 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [17/Sep/2024:09:28:02] ENGINE Error in HTTPServer.serve 2024-09-17T09:28:37.559 INFO:journalctl@ceph.mon.c.smithi029.stdout: self.do_handshake() 2024-09-17T09:28:37.560 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:28:37.560 INFO:journalctl@ceph.mon.c.smithi029.stdout: self._sslobj.do_handshake() 2024-09-17T09:28:37.560 INFO:journalctl@ceph.mon.c.smithi029.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:28:37.560 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.560 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring iscsi.foo.smithi029.gellkh (dependencies changed)... 2024-09-17T09:28:37.560 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:28:37.560 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T09:28:37.560 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Reconfiguring daemon prometheus.a on smithi103 2024-09-17T09:28:37.560 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Upgrade: Updating mgr.x 2024-09-17T09:28:37.560 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Deploying daemon mgr.x on smithi103 2024-09-17T09:28:37.560 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.560 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Upgrade: Setting container_image for all mgr 2024-09-17T09:28:37.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Upgrade: It appears safe to stop mon.a 2024-09-17T09:28:37.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Upgrade: Updating mon.a 2024-09-17T09:28:37.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Deploying daemon mon.a on smithi029 2024-09-17T09:28:37.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: fsmap 2024-09-17T09:28:37.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: osdmap e77: 8 total, 8 up, 8 in 2024-09-17T09:28:37.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mgrmap e26: y(active, since 41s), standbys: x 2024-09-17T09:28:37.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:28:37.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:37.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:37.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:37.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mgrmap e27: y(active, since 41s), standbys: x 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mgrmap e28: y(active, since 42s), standbys: x 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Standby manager daemon x restarted 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Standby manager daemon x started 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mgrmap e29: y(active, since 45s), standbys: x 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Active manager daemon y restarted 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Activating manager daemon y 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mgrmap e30: y(active, starting, since 0.00714194s), standbys: x 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Manager daemon y is now available 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mgrmap e31: y(active, since 1.01065s), standbys: x 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v3: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Cluster is now healthy 2024-09-17T09:28:37.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mgrmap e32: y(active, since 2s), standbys: x 2024-09-17T09:28:37.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [17/Sep/2024:09:28:02] ENGINE Bus STARTING 2024-09-17T09:28:37.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [17/Sep/2024:09:28:02] ENGINE Serving on http://172.21.15.29:8765 2024-09-17T09:28:37.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [17/Sep/2024:09:28:02] ENGINE Serving on https://172.21.15.29:7150 2024-09-17T09:28:37.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [17/Sep/2024:09:28:02] ENGINE Bus STARTED 2024-09-17T09:28:37.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [17/Sep/2024:09:28:02] ENGINE Error in HTTPServer.serve 2024-09-17T09:28:37.563 INFO:journalctl@ceph.mon.a.smithi029.stdout: Traceback (most recent call last): 2024-09-17T09:28:37.563 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:28:37.563 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:28:37.564 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:28:37.564 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._run(expiration_interval) 2024-09-17T09:28:37.564 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:28:37.564 INFO:journalctl@ceph.mon.a.smithi029.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:28:37.564 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:28:37.564 INFO:journalctl@ceph.mon.a.smithi029.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:28:37.564 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:28:37.564 INFO:journalctl@ceph.mon.a.smithi029.stdout: s = self.context.wrap_socket( 2024-09-17T09:28:37.564 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:28:37.564 INFO:journalctl@ceph.mon.a.smithi029.stdout: return self.sslsocket_class._create( 2024-09-17T09:28:37.565 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:28:37.565 INFO:journalctl@ceph.mon.a.smithi029.stdout: self.do_handshake() 2024-09-17T09:28:37.565 INFO:journalctl@ceph.mon.a.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:28:37.565 INFO:journalctl@ceph.mon.a.smithi029.stdout: self._sslobj.do_handshake() 2024-09-17T09:28:37.565 INFO:journalctl@ceph.mon.a.smithi029.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:28:37.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mgrmap e33: y(active, since 4s), standbys: x 2024-09-17T09:28:37.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-17T09:28:37.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:37.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:37.565 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:37.566 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/1781687977' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:37.566 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:37.566 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T09:28:37.566 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='client.44190 172.21.15.29:0/197166824' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:28:37.566 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.566 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.566 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:37.566 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:28:37.566 INFO:journalctl@ceph.mon.c.smithi029.stdout: Traceback (most recent call last): 2024-09-17T09:28:37.566 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T09:28:37.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: self._connections.run(self.expiration_interval) 2024-09-17T09:28:37.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T09:28:37.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: self._run(expiration_interval) 2024-09-17T09:28:37.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T09:28:37.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T09:28:37.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T09:28:37.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T09:28:37.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T09:28:37.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: s = self.context.wrap_socket( 2024-09-17T09:28:37.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T09:28:37.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: return self.sslsocket_class._create( 2024-09-17T09:28:37.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T09:28:37.567 INFO:journalctl@ceph.mon.c.smithi029.stdout: self.do_handshake() 2024-09-17T09:28:37.568 INFO:journalctl@ceph.mon.c.smithi029.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T09:28:37.568 INFO:journalctl@ceph.mon.c.smithi029.stdout: self._sslobj.do_handshake() 2024-09-17T09:28:37.568 INFO:journalctl@ceph.mon.c.smithi029.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T09:28:37.568 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v5: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.568 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mgrmap e33: y(active, since 4s), standbys: x 2024-09-17T09:28:37.568 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.568 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.568 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v6: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-17T09:28:37.568 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:37.568 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:37.568 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:37.569 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/1781687977' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:37.569 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:37.569 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v7: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-17T09:28:37.569 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='client.44190 172.21.15.29:0/197166824' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:28:37.569 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.569 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.569 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:37.569 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:28:37.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-17T09:28:37.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:28:37.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:37.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T09:28:37.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T09:28:37.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Upgrade: It appears safe to stop mon.c 2024-09-17T09:28:37.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/871367148' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:37.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:28:37.570 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='client.44214 172.21.15.29:0/552312000' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]: access denied 2024-09-17T09:28:37.571 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Upgrade: Updating mon.c 2024-09-17T09:28:37.571 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.571 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:37.571 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:37.571 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:37.571 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Deploying daemon mon.c on smithi029 2024-09-17T09:28:37.571 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:28:37.571 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:28:37.571 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:28:37.571 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:28:37.571 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:28:37.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:28:37.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mon.c calling monitor election 2024-09-17T09:28:37.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mon.a calling monitor election 2024-09-17T09:28:37.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T09:28:37.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:28:37.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: fsmap 2024-09-17T09:28:37.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T09:28:37.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mgrmap e33: y(active, since 16s), standbys: x 2024-09-17T09:28:37.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:28:37.572 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:37.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v8: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-17T09:28:37.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:28:37.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:37.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T09:28:37.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T09:28:37.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Upgrade: It appears safe to stop mon.c 2024-09-17T09:28:37.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/871367148' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:37.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:28:37.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='client.44214 172.21.15.29:0/552312000' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]: access denied 2024-09-17T09:28:37.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Upgrade: Updating mon.c 2024-09-17T09:28:37.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:37.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:37.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:37.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Deploying daemon mon.c on smithi029 2024-09-17T09:28:37.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v9: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T09:28:37.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v10: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:28:37.574 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:28:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:28:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:28:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v11: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T09:28:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mon.c calling monitor election 2024-09-17T09:28:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mon.a calling monitor election 2024-09-17T09:28:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T09:28:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:28:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: fsmap 2024-09-17T09:28:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T09:28:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mgrmap e33: y(active, since 16s), standbys: x 2024-09-17T09:28:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:28:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:37.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:37.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:37.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 91 B/s rd, 0 op/s 2024-09-17T09:28:37.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 91 B/s rd, 0 op/s 2024-09-17T09:28:37.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='client.44232 172.21.15.29:0/840260197' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='client.54121 172.21.15.29:0/3440964050' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:28:37.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:28:37.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:37.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T09:28:37.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T09:28:37.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Upgrade: It appears safe to stop mon.b 2024-09-17T09:28:37.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Upgrade: Updating mon.b 2024-09-17T09:28:37.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:37.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 ' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:37.578 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:37.578 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:37.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:37.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v12: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 91 B/s rd, 0 op/s 2024-09-17T09:28:37.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v13: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 91 B/s rd, 0 op/s 2024-09-17T09:28:37.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v14: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:37.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.579 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v15: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='client.44232 172.21.15.29:0/840260197' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:37.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v16: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:37.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:28:37.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:37.580 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v17: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.581 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v18: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:37.581 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.581 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='client.54121 172.21.15.29:0/3440964050' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:28:37.581 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:28:37.581 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:37.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T09:28:37.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T09:28:37.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Upgrade: It appears safe to stop mon.b 2024-09-17T09:28:37.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Upgrade: Updating mon.b 2024-09-17T09:28:37.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' 2024-09-17T09:28:37.582 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:37.583 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 ' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:37.583 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:37.583 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Deploying daemon mon.b on smithi103 2024-09-17T09:28:37.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T09:28:37.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:28:37.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:28:37.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:28:37.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mon.b calling monitor election 2024-09-17T09:28:37.584 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mon.c calling monitor election 2024-09-17T09:28:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); 1/3 mons down, quorum a,c 2024-09-17T09:28:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-17T09:28:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mon.b (rank 2) addr [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] is down (out of quorum) 2024-09-17T09:28:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:28:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:28:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:28:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mon.a calling monitor election 2024-09-17T09:28:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T09:28:37.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: monmap e4: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:28:37.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: fsmap 2024-09-17T09:28:37.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T09:28:37.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: mgrmap e33: y(active, since 36s), standbys: x 2024-09-17T09:28:37.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-17T09:28:37.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:28:37.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:37.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:37.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[58326]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:37.586 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Deploying daemon mon.b on smithi103 2024-09-17T09:28:37.586 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: pgmap v19: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mon.b calling monitor election 2024-09-17T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mon.c calling monitor election 2024-09-17T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); 1/3 mons down, quorum a,c 2024-09-17T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-17T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mon.b (rank 2) addr [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] is down (out of quorum) 2024-09-17T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T09:28:37.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T09:28:37.588 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T09:28:37.588 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mon.a calling monitor election 2024-09-17T09:28:37.588 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T09:28:37.588 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: monmap e4: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-17T09:28:37.588 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: fsmap 2024-09-17T09:28:37.588 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T09:28:37.588 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: mgrmap e33: y(active, since 36s), standbys: x 2024-09-17T09:28:37.588 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-17T09:28:37.588 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:28:37.588 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:37.588 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:37.588 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:37 smithi029 ceph-mon[60489]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:37.974 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:28:38.421 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:38 smithi103 ceph-mon[49967]: pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:38.421 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:38 smithi103 ceph-mon[49967]: from='client.54139 172.21.15.29:0/3694670236' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:38.421 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:38 smithi103 ceph-mon[49967]: from='client.44262 172.21.15.29:0/3678317472' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:28:38.543 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:38 smithi029 ceph-mon[60489]: pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:38.543 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:38 smithi029 ceph-mon[60489]: from='client.54139 172.21.15.29:0/3694670236' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:38.543 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:38 smithi029 ceph-mon[60489]: from='client.44262 172.21.15.29:0/3678317472' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:28:38.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:38 smithi029 ceph-mon[58326]: pgmap v24: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:38.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:38 smithi029 ceph-mon[58326]: from='client.54139 172.21.15.29:0/3694670236' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:38.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:38 smithi029 ceph-mon[58326]: from='client.44262 172.21.15.29:0/3678317472' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:28:38.644 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:28:38.644 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (2m) 18s ago 6m 15.0M - 0.20.0 0881eb8f169f 7bd158a6d65c 2024-09-17T09:28:38.644 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (105s) 34s ago 6m 28.8M - 6.7.4 557c83e11646 e361816f7c14 2024-09-17T09:28:38.644 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 unknown 18s ago 5m - - 2024-09-17T09:28:38.644 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (61s) 34s ago 10m 434M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:28:38.644 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (98s) 18s ago 12m 474M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:28:38.644 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (46s) 18s ago 12m 40.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:28:38.645 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 starting - - - 2048M 2024-09-17T09:28:38.645 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (23s) 18s ago 11m 25.3M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 0d3d1a65e2c9 2024-09-17T09:28:38.645 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (2m) 18s ago 7m 15.0M - 0.18.1 e5a616e4b9cf 34bfa263169f 2024-09-17T09:28:38.645 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (117s) 34s ago 7m 6777k - 0.18.1 e5a616e4b9cf 1adf1b5baad1 2024-09-17T09:28:38.645 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (10m) 18s ago 10m 48.3M 4096M 16.2.0 24ecd6d5f14c 991731ce08b5 2024-09-17T09:28:38.645 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (10m) 18s ago 10m 46.2M 4096M 16.2.0 24ecd6d5f14c d5bce63fadc8 2024-09-17T09:28:38.645 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (9m) 18s ago 9m 43.8M 4096M 16.2.0 24ecd6d5f14c e9678d2f1685 2024-09-17T09:28:38.645 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (9m) 18s ago 9m 49.8M 4096M 16.2.0 24ecd6d5f14c dcf7122a7d99 2024-09-17T09:28:38.645 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (9m) 34s ago 9m 49.0M 4096M 16.2.0 24ecd6d5f14c 048c13025d32 2024-09-17T09:28:38.645 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (8m) 34s ago 8m 47.5M 4096M 16.2.0 24ecd6d5f14c 10a64919fbc9 2024-09-17T09:28:38.645 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (8m) 34s ago 8m 46.1M 4096M 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:28:38.645 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (8m) 34s ago 8m 52.3M 4096M 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:28:38.645 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 unknown 34s ago 7m - - 2024-09-17T09:28:38.646 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (6m) 18s ago 6m 40.6M - 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:28:38.646 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (6m) 34s ago 6m 40.4M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:28:38.646 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (6m) 18s ago 6m 38.7M - 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:28:38.646 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (6m) 34s ago 6m 37.4M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:28:39.056 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:28:39.057 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:28:39.057 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:28:39.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:28:39.057 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:28:39.057 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:28:39.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:28:39.057 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:28:39.057 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T09:28:39.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:28:39.057 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:28:39.057 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:28:39.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:28:39.058 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:28:39.058 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 12, 2024-09-17T09:28:39.058 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 5 2024-09-17T09:28:39.058 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:28:39.058 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:28:39.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:39 smithi029 ceph-mon[58326]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:39.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:39 smithi029 ceph-mon[58326]: from='client.44286 172.21.15.29:0/115144380' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:39 smithi029 ceph-mon[60489]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:39 smithi029 ceph-mon[60489]: from='client.44286 172.21.15.29:0/115144380' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:39.399 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:28:39.399 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:28:39.399 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:28:39.399 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:28:39.399 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [ 2024-09-17T09:28:39.399 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr" 2024-09-17T09:28:39.399 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:28:39.399 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "4/23 daemons upgraded", 2024-09-17T09:28:39.400 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Currently upgrading mon daemons", 2024-09-17T09:28:39.400 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:28:39.400 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:28:39.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:39 smithi103 ceph-mon[49967]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:39.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:39 smithi103 ceph-mon[49967]: from='client.44286 172.21.15.29:0/115144380' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:28:39.797 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:28:39 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:28:39.535+0000 7fe73b7e7640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-17T09:28:39.803 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:28:39.803 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:28:39.803 INFO:teuthology.orchestra.run.smithi029.stdout: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:28:39.804 INFO:teuthology.orchestra.run.smithi029.stdout: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:28:40.444 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:40 smithi103 ceph-mon[49967]: from='client.54157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:40.444 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:40 smithi103 ceph-mon[49967]: pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:40.444 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:40 smithi103 ceph-mon[49967]: from='client.44280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:40.444 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:40 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:40.444 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:40 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:40.444 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:40 smithi103 ceph-mon[49967]: from='client.44298 172.21.15.29:0/18774635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:28:40.444 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:40 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:40.445 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:40 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:40.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[58326]: from='client.54157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:40.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[58326]: pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:40.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[58326]: from='client.44280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:40.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:40.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:40.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[58326]: from='client.44298 172.21.15.29:0/18774635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:28:40.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:40.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:40.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[60489]: from='client.54157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:40.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[60489]: pgmap v25: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:40.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[60489]: from='client.44280 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:40.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:40.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:40.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[60489]: from='client.44298 172.21.15.29:0/18774635' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:28:40.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:40.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:40 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:41.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:40 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 6. 2024-09-17T09:28:41.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:40 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:41.119 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:40 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:28:41.449 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-mon[49967]: from='client.44292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:41.450 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 podman[51025]: 2024-09-17 09:28:41.022191436 +0000 UTC m=+0.017128341 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:28:41.450 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 podman[51025]: 2024-09-17 09:28:41.152039771 +0000 UTC m=+0.146976664 container create 46179a62e2da9d3ade67a178e51de57a09414d81c25633879d09e171e87dce26 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:41.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:41 smithi029 ceph-mon[58326]: from='client.44292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:41.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:41 smithi029 ceph-mon[60489]: from='client.44292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:28:41.859 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 podman[51025]: 2024-09-17 09:28:41.810753055 +0000 UTC m=+0.805689950 container init 46179a62e2da9d3ade67a178e51de57a09414d81c25633879d09e171e87dce26 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:41.859 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 podman[51025]: 2024-09-17 09:28:41.814284813 +0000 UTC m=+0.809221716 container start 46179a62e2da9d3ade67a178e51de57a09414d81c25633879d09e171e87dce26 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:41.859 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.845Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:28:41.859 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.845Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:28:41.859 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.845Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:28:41.859 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.845Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:28:41.859 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.845Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:28:41.859 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.845Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:28:41.859 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.846Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:28:41.859 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.846Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:28:41.859 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.848Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:28:41.859 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.848Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=13 2024-09-17T09:28:41.860 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.855Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=13 2024-09-17T09:28:41.860 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.855Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=13 2024-09-17T09:28:41.860 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.855Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=13 2024-09-17T09:28:41.860 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.855Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=13 2024-09-17T09:28:41.860 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.856Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=13 2024-09-17T09:28:41.860 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.856Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=13 2024-09-17T09:28:41.860 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.856Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=13 2024-09-17T09:28:41.860 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.856Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=13 2024-09-17T09:28:41.860 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.856Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=13 2024-09-17T09:28:41.860 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.856Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=13 2024-09-17T09:28:42.185 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.858Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=13 2024-09-17T09:28:42.185 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.858Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=12 maxSegment=13 2024-09-17T09:28:42.185 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.859Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=13 maxSegment=13 2024-09-17T09:28:42.185 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.859Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=10.757669ms 2024-09-17T09:28:42.186 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.861Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:28:42.186 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.861Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:28:42.186 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.861Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:28:42.186 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.861Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:28:42.186 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.861Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:28:42.186 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.861Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:28:42.186 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.861Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:28:42.186 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.861Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:28:42.186 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.861Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:28:42.186 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.861Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:28:42.186 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.861Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:28:42.187 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.861Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:28:42.187 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=info ts=2024-09-17T09:28:41.861Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:28:42.187 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51040]: level=error ts=2024-09-17T09:28:41.861Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:28:42.187 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 bash[51025]: 46179a62e2da9d3ade67a178e51de57a09414d81c25633879d09e171e87dce26 2024-09-17T09:28:42.187 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:42.187 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:41 smithi103 podman[51082]: 2024-09-17 09:28:41.939284466 +0000 UTC m=+0.070345274 container died 46179a62e2da9d3ade67a178e51de57a09414d81c25633879d09e171e87dce26 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:42.484 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:42 smithi103 ceph-mon[49967]: pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:42.485 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:42 smithi103 podman[51082]: 2024-09-17 09:28:42.484274188 +0000 UTC m=+0.615334981 container remove 46179a62e2da9d3ade67a178e51de57a09414d81c25633879d09e171e87dce26 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:42.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:42 smithi029 ceph-mon[58326]: pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:42.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:42 smithi029 ceph-mon[60489]: pgmap v26: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:42.754 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:42 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:28:42.754 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:42 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:28:44.242 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:44 smithi103 ceph-mon[49967]: pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:44.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:44 smithi029 ceph-mon[58326]: pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:44.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:44 smithi029 ceph-mon[60489]: pgmap v27: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:45.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:45.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:45.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:45.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:28:45.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:45.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[58326]: pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2024-09-17T09:28:45.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[58326]: pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 105 B/s rd, 0 op/s 2024-09-17T09:28:45.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:45.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[58326]: Reconfiguring mon.a (monmap changed)... 2024-09-17T09:28:45.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:45.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:45.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:45.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[58326]: Reconfiguring daemon mon.a on smithi029 2024-09-17T09:28:45.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:45.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:45.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:45.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:28:45.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:45.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[60489]: pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2024-09-17T09:28:45.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[60489]: pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 105 B/s rd, 0 op/s 2024-09-17T09:28:45.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:45.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[60489]: Reconfiguring mon.a (monmap changed)... 2024-09-17T09:28:45.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:45.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:45.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:45.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:45 smithi029 ceph-mon[60489]: Reconfiguring daemon mon.a on smithi029 2024-09-17T09:28:45.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:45 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:45.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:45 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:45.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:45 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:45.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:45 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:28:45.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:45 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:28:45.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:45 smithi103 ceph-mon[49967]: pgmap v28: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 87 B/s rd, 0 op/s 2024-09-17T09:28:45.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:45 smithi103 ceph-mon[49967]: pgmap v29: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 105 B/s rd, 0 op/s 2024-09-17T09:28:45.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:45 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:45.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:45 smithi103 ceph-mon[49967]: Reconfiguring mon.a (monmap changed)... 2024-09-17T09:28:45.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:45 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:45.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:45 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:45.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:45 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:45.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:45 smithi103 ceph-mon[49967]: Reconfiguring daemon mon.a on smithi029 2024-09-17T09:28:47.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:47.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:47.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:46 smithi103 ceph-mon[49967]: Reconfiguring mgr.y (monmap changed)... 2024-09-17T09:28:47.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:28:47.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:28:47.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:47.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:46 smithi103 ceph-mon[49967]: Reconfiguring daemon mgr.y on smithi029 2024-09-17T09:28:47.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:47.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:47.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[58326]: Reconfiguring mgr.y (monmap changed)... 2024-09-17T09:28:47.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:28:47.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:28:47.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:47.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[58326]: Reconfiguring daemon mgr.y on smithi029 2024-09-17T09:28:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[60489]: Reconfiguring mgr.y (monmap changed)... 2024-09-17T09:28:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:28:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:28:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:46 smithi029 ceph-mon[60489]: Reconfiguring daemon mgr.y on smithi029 2024-09-17T09:28:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[60489]: pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 105 B/s rd, 0 op/s 2024-09-17T09:28:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[60489]: Reconfiguring mon.c (monmap changed)... 2024-09-17T09:28:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[60489]: Reconfiguring daemon mon.c on smithi029 2024-09-17T09:28:48.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[58326]: pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 105 B/s rd, 0 op/s 2024-09-17T09:28:48.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:48.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:48.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[58326]: Reconfiguring mon.c (monmap changed)... 2024-09-17T09:28:48.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:48.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:48.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:48.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:48 smithi029 ceph-mon[58326]: Reconfiguring daemon mon.c on smithi029 2024-09-17T09:28:48.464 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:48 smithi103 ceph-mon[49967]: pgmap v30: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 105 B/s rd, 0 op/s 2024-09-17T09:28:48.464 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:48.464 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:48.464 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:48 smithi103 ceph-mon[49967]: Reconfiguring mon.c (monmap changed)... 2024-09-17T09:28:48.465 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:28:48.465 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:28:48.465 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:48.465 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:48 smithi103 ceph-mon[49967]: Reconfiguring daemon mon.c on smithi029 2024-09-17T09:28:49.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:49 smithi103 ceph-mon[49967]: pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 105 B/s rd, 0 op/s 2024-09-17T09:28:49.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:49 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:49.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:49 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:49.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:49 smithi103 ceph-mon[49967]: Reconfiguring osd.0 (monmap changed)... 2024-09-17T09:28:49.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:49 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T09:28:49.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:49 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:49.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:49 smithi103 ceph-mon[49967]: Reconfiguring daemon osd.0 on smithi029 2024-09-17T09:28:49.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[58326]: pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 105 B/s rd, 0 op/s 2024-09-17T09:28:49.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:49.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:49.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[58326]: Reconfiguring osd.0 (monmap changed)... 2024-09-17T09:28:49.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T09:28:49.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:49.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[58326]: Reconfiguring daemon osd.0 on smithi029 2024-09-17T09:28:49.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[60489]: pgmap v31: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 105 B/s rd, 0 op/s 2024-09-17T09:28:49.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:49.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:49.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[60489]: Reconfiguring osd.0 (monmap changed)... 2024-09-17T09:28:49.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T09:28:49.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:49.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:49 smithi029 ceph-mon[60489]: Reconfiguring daemon osd.0 on smithi029 2024-09-17T09:28:51.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:51 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:51.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:51 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:51.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:51 smithi029 ceph-mon[58326]: Reconfiguring osd.1 (monmap changed)... 2024-09-17T09:28:51.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:51 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T09:28:51.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:51 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:51.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:51 smithi029 ceph-mon[58326]: Reconfiguring daemon osd.1 on smithi029 2024-09-17T09:28:51.091 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:51 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:51.091 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:51 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:51.091 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:51 smithi029 ceph-mon[60489]: Reconfiguring osd.1 (monmap changed)... 2024-09-17T09:28:51.091 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:51 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T09:28:51.092 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:51 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:51.092 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:51 smithi029 ceph-mon[60489]: Reconfiguring daemon osd.1 on smithi029 2024-09-17T09:28:51.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:51 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:51.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:51 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:51.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:51 smithi103 ceph-mon[49967]: Reconfiguring osd.1 (monmap changed)... 2024-09-17T09:28:51.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:51 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T09:28:51.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:51 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:51.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:51 smithi103 ceph-mon[49967]: Reconfiguring daemon osd.1 on smithi029 2024-09-17T09:28:52.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:52 smithi029 ceph-mon[58326]: pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:52.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:52 smithi029 ceph-mon[58326]: from='client.44316 172.21.15.29:0/490972555' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:52.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:52 smithi029 ceph-mon[58326]: from='client.44322 172.21.15.29:0/1696007972' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:28:52.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:52 smithi029 ceph-mon[60489]: pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:52.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:52 smithi029 ceph-mon[60489]: from='client.44316 172.21.15.29:0/490972555' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:52.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:52 smithi029 ceph-mon[60489]: from='client.44322 172.21.15.29:0/1696007972' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:28:52.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:52 smithi103 ceph-mon[49967]: pgmap v32: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:52.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:52 smithi103 ceph-mon[49967]: from='client.44316 172.21.15.29:0/490972555' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:28:52.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:52 smithi103 ceph-mon[49967]: from='client.44322 172.21.15.29:0/1696007972' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:28:53.063 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:52 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 7. 2024-09-17T09:28:53.063 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:52 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:53.064 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:52 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:28:53.064 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:52 smithi103 podman[51236]: 2024-09-17 09:28:52.844281869 +0000 UTC m=+0.017359625 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:28:53.064 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:52 smithi103 podman[51236]: 2024-09-17 09:28:52.943361377 +0000 UTC m=+0.116439121 container create bc79cef42b9b78abc9619df68053c9181eeb262948af7895ef59f3b97e27157b (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:53.315 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:53.315 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:53.315 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-mon[49967]: Reconfiguring osd.2 (monmap changed)... 2024-09-17T09:28:53.315 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T09:28:53.315 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:53.315 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-mon[49967]: Reconfiguring daemon osd.2 on smithi029 2024-09-17T09:28:53.316 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 podman[51236]: 2024-09-17 09:28:53.187175595 +0000 UTC m=+0.360253338 container init bc79cef42b9b78abc9619df68053c9181eeb262948af7895ef59f3b97e27157b (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:53.316 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 podman[51236]: 2024-09-17 09:28:53.190140372 +0000 UTC m=+0.363218127 container start bc79cef42b9b78abc9619df68053c9181eeb262948af7895ef59f3b97e27157b (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:53.316 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.220Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:28:53.316 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.220Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:28:53.316 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.220Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:28:53.316 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.220Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:28:53.316 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.220Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:28:53.316 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.220Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:28:53.316 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.221Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:28:53.316 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.221Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:28:53.316 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.223Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.223Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=14 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.231Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=14 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.231Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=14 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.231Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=14 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.232Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=14 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.232Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=14 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.232Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=14 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.232Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=14 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.232Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=14 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.232Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=14 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.233Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=14 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.233Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=14 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.234Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=12 maxSegment=14 2024-09-17T09:28:53.317 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.235Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=13 maxSegment=14 2024-09-17T09:28:53.318 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.235Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=14 maxSegment=14 2024-09-17T09:28:53.318 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.235Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=11.727471ms 2024-09-17T09:28:53.318 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.236Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:28:53.318 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.236Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:28:53.318 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.236Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:28:53.318 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.237Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:28:53.318 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.237Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:28:53.318 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.237Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:28:53.318 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.237Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:28:53.318 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.237Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:28:53.319 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.237Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:28:53.319 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.237Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:28:53.319 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.237Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:28:53.319 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.237Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:28:53.319 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=info ts=2024-09-17T09:28:53.237Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:28:53.319 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51251]: level=error ts=2024-09-17T09:28:53.237Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:28:53.319 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 bash[51236]: bc79cef42b9b78abc9619df68053c9181eeb262948af7895ef59f3b97e27157b 2024-09-17T09:28:53.319 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:28:53.319 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 podman[51268]: 2024-09-17 09:28:53.314409396 +0000 UTC m=+0.069601234 container died bc79cef42b9b78abc9619df68053c9181eeb262948af7895ef59f3b97e27157b (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:53.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:53 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:53.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:53 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:53.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:53 smithi029 ceph-mon[60489]: Reconfiguring osd.2 (monmap changed)... 2024-09-17T09:28:53.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:53 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T09:28:53.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:53 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:53.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:53 smithi029 ceph-mon[60489]: Reconfiguring daemon osd.2 on smithi029 2024-09-17T09:28:53.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:53 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:53.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:53 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:53.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:53 smithi029 ceph-mon[58326]: Reconfiguring osd.2 (monmap changed)... 2024-09-17T09:28:53.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:53 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T09:28:53.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:53 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:53.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:53 smithi029 ceph-mon[58326]: Reconfiguring daemon osd.2 on smithi029 2024-09-17T09:28:54.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:54 smithi103 ceph-mon[49967]: pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:54.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 podman[51268]: 2024-09-17 09:28:53.941723646 +0000 UTC m=+0.696915472 container remove bc79cef42b9b78abc9619df68053c9181eeb262948af7895ef59f3b97e27157b (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:28:54.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:28:54.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:28:53 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:28:54.383 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:54 smithi029 ceph-mon[60489]: pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:54.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:54 smithi029 ceph-mon[58326]: pgmap v33: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:56.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[58326]: pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:56.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:56.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:56.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[58326]: Reconfiguring osd.3 (monmap changed)... 2024-09-17T09:28:56.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T09:28:56.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:56.121 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[58326]: Reconfiguring daemon osd.3 on smithi029 2024-09-17T09:28:56.121 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[60489]: pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:56.121 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:56.121 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:56.121 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[60489]: Reconfiguring osd.3 (monmap changed)... 2024-09-17T09:28:56.121 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T09:28:56.121 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:56.121 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:55 smithi029 ceph-mon[60489]: Reconfiguring daemon osd.3 on smithi029 2024-09-17T09:28:56.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:55 smithi103 ceph-mon[49967]: pgmap v34: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:28:56.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:55 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:56.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:55 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:56.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:55 smithi103 ceph-mon[49967]: Reconfiguring osd.3 (monmap changed)... 2024-09-17T09:28:56.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:55 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T09:28:56.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:55 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:56.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:55 smithi103 ceph-mon[49967]: Reconfiguring daemon osd.3 on smithi029 2024-09-17T09:28:57.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:57 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:57.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:57 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:57.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:57 smithi103 ceph-mon[49967]: Reconfiguring rgw.foo.smithi029.tyykln (monmap changed)... 2024-09-17T09:28:57.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:57 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi029.tyykln", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:28:57.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:57 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:57.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:57 smithi103 ceph-mon[49967]: Reconfiguring daemon rgw.foo.smithi029.tyykln on smithi029 2024-09-17T09:28:57.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:57 smithi103 ceph-mon[49967]: pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:57.510 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:57.510 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:57.510 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[58326]: Reconfiguring rgw.foo.smithi029.tyykln (monmap changed)... 2024-09-17T09:28:57.510 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi029.tyykln", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:28:57.510 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:57.510 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[58326]: Reconfiguring daemon rgw.foo.smithi029.tyykln on smithi029 2024-09-17T09:28:57.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[58326]: pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:57.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:57.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:57.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[60489]: Reconfiguring rgw.foo.smithi029.tyykln (monmap changed)... 2024-09-17T09:28:57.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi029.tyykln", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:28:57.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:57.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[60489]: Reconfiguring daemon rgw.foo.smithi029.tyykln on smithi029 2024-09-17T09:28:57.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:57 smithi029 ceph-mon[60489]: pgmap v35: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:28:58.836 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:58 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:58.836 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:58 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:58.836 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:58 smithi029 ceph-mon[60489]: Reconfiguring rgw.smpl.smithi029.prstdw (monmap changed)... 2024-09-17T09:28:58.836 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:58 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi029.prstdw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:28:58.836 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:58 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:58.836 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:58 smithi029 ceph-mon[60489]: Reconfiguring daemon rgw.smpl.smithi029.prstdw on smithi029 2024-09-17T09:28:58.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:58 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:58.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:58 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:58.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:58 smithi029 ceph-mon[58326]: Reconfiguring rgw.smpl.smithi029.prstdw (monmap changed)... 2024-09-17T09:28:58.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:58 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi029.prstdw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:28:58.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:58 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:58.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:58 smithi029 ceph-mon[58326]: Reconfiguring daemon rgw.smpl.smithi029.prstdw on smithi029 2024-09-17T09:28:59.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:58 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:59.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:58 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:28:59.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:58 smithi103 ceph-mon[49967]: Reconfiguring rgw.smpl.smithi029.prstdw (monmap changed)... 2024-09-17T09:28:59.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:58 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi029.prstdw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:28:59.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:58 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:28:59.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:58 smithi103 ceph-mon[49967]: Reconfiguring daemon rgw.smpl.smithi029.prstdw on smithi029 2024-09-17T09:29:00.234 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:59 smithi103 ceph-mon[49967]: pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:00.235 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:59 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:00.235 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:59 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:00.235 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:59 smithi103 ceph-mon[49967]: Reconfiguring mon.b (monmap changed)... 2024-09-17T09:29:00.235 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:59 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:29:00.235 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:59 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:29:00.235 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:59 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:00.235 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:28:59 smithi103 ceph-mon[49967]: Reconfiguring daemon mon.b on smithi103 2024-09-17T09:29:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[58326]: pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[58326]: Reconfiguring mon.b (monmap changed)... 2024-09-17T09:29:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:29:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:29:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[58326]: Reconfiguring daemon mon.b on smithi103 2024-09-17T09:29:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[60489]: pgmap v36: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[60489]: Reconfiguring mon.b (monmap changed)... 2024-09-17T09:29:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T09:29:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T09:29:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:28:59 smithi029 ceph-mon[60489]: Reconfiguring daemon mon.b on smithi103 2024-09-17T09:29:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[58326]: pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:01.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:01.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[58326]: Reconfiguring mgr.x (monmap changed)... 2024-09-17T09:29:01.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:29:01.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:29:01.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:01.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[58326]: Reconfiguring daemon mgr.x on smithi103 2024-09-17T09:29:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[60489]: pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[60489]: Reconfiguring mgr.x (monmap changed)... 2024-09-17T09:29:01.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:29:01.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:29:01.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:01.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:01 smithi029 ceph-mon[60489]: Reconfiguring daemon mgr.x on smithi103 2024-09-17T09:29:01.559 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:01 smithi103 ceph-mon[49967]: pgmap v37: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:01.559 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:01.559 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:01.559 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:01 smithi103 ceph-mon[49967]: Reconfiguring mgr.x (monmap changed)... 2024-09-17T09:29:01.559 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T09:29:01.559 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T09:29:01.559 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:01.559 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:01 smithi103 ceph-mon[49967]: Reconfiguring daemon mgr.x on smithi103 2024-09-17T09:29:02.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:02 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:02.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:02 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:02.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:02 smithi103 ceph-mon[49967]: Reconfiguring osd.4 (monmap changed)... 2024-09-17T09:29:02.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:02 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T09:29:02.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:02 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:02.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:02 smithi103 ceph-mon[49967]: Reconfiguring daemon osd.4 on smithi103 2024-09-17T09:29:03.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:02 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:03.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:02 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:03.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:02 smithi029 ceph-mon[58326]: Reconfiguring osd.4 (monmap changed)... 2024-09-17T09:29:03.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:02 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T09:29:03.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:02 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:03.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:02 smithi029 ceph-mon[58326]: Reconfiguring daemon osd.4 on smithi103 2024-09-17T09:29:03.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:02 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:03.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:02 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:03.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:02 smithi029 ceph-mon[60489]: Reconfiguring osd.4 (monmap changed)... 2024-09-17T09:29:03.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:02 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T09:29:03.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:02 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:03.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:02 smithi029 ceph-mon[60489]: Reconfiguring daemon osd.4 on smithi103 2024-09-17T09:29:04.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 8. 2024-09-17T09:29:04.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:29:04.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:29:04.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:03 smithi103 ceph-mon[49967]: pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:04.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:03 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:04.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:03 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:04.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:03 smithi103 ceph-mon[49967]: Reconfiguring osd.5 (monmap changed)... 2024-09-17T09:29:04.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:03 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T09:29:04.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:03 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:04.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:03 smithi103 ceph-mon[49967]: Reconfiguring daemon osd.5 on smithi103 2024-09-17T09:29:04.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[60489]: pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:04.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:04.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:04.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[60489]: Reconfiguring osd.5 (monmap changed)... 2024-09-17T09:29:04.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T09:29:04.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:04.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[60489]: Reconfiguring daemon osd.5 on smithi103 2024-09-17T09:29:04.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[58326]: pgmap v38: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:04.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:04.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:04.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[58326]: Reconfiguring osd.5 (monmap changed)... 2024-09-17T09:29:04.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T09:29:04.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:04.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:03 smithi029 ceph-mon[58326]: Reconfiguring daemon osd.5 on smithi103 2024-09-17T09:29:04.552 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 podman[51835]: 2024-09-17 09:29:04.431641475 +0000 UTC m=+0.017296458 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:29:04.552 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 podman[51835]: 2024-09-17 09:29:04.551654281 +0000 UTC m=+0.137309239 container create cd4238c9061fbd0fe2850bd03af1e6a84d2000bcc23dddbbe8334dc3b97d9296 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:29:04.803 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 podman[51835]: 2024-09-17 09:29:04.770140913 +0000 UTC m=+0.355795883 container init cd4238c9061fbd0fe2850bd03af1e6a84d2000bcc23dddbbe8334dc3b97d9296 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:29:04.803 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 podman[51835]: 2024-09-17 09:29:04.772971427 +0000 UTC m=+0.358626386 container start cd4238c9061fbd0fe2850bd03af1e6a84d2000bcc23dddbbe8334dc3b97d9296 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:29:05.159 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.802Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:29:05.159 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.802Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:29:05.160 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.802Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:29:05.160 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.802Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:29:05.160 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.802Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:29:05.160 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.802Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:29:05.160 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.803Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:29:05.160 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.803Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:29:05.160 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.805Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:29:05.160 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.806Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=15 2024-09-17T09:29:05.160 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.813Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=15 2024-09-17T09:29:05.160 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.813Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=15 2024-09-17T09:29:05.160 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.813Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=15 2024-09-17T09:29:05.160 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.813Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=15 2024-09-17T09:29:05.161 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.813Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=15 2024-09-17T09:29:05.161 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.813Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=15 2024-09-17T09:29:05.161 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.814Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=15 2024-09-17T09:29:05.161 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.814Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=15 2024-09-17T09:29:05.161 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.814Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=15 2024-09-17T09:29:05.161 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.814Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=15 2024-09-17T09:29:05.161 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.814Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=15 2024-09-17T09:29:05.162 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.815Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=12 maxSegment=15 2024-09-17T09:29:05.162 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.816Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=13 maxSegment=15 2024-09-17T09:29:05.162 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.816Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=14 maxSegment=15 2024-09-17T09:29:05.162 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.816Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=15 maxSegment=15 2024-09-17T09:29:05.162 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.817Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=11.223991ms 2024-09-17T09:29:05.162 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.818Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:29:05.162 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.818Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:29:05.162 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.818Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:29:05.162 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.819Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:29:05.162 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.819Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:29:05.162 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.819Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:29:05.162 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.819Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:29:05.163 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.819Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:29:05.163 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.819Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:29:05.163 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.819Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:29:05.163 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.819Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:29:05.163 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.819Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:29:05.163 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=info ts=2024-09-17T09:29:04.819Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:29:05.163 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[51915]: level=error ts=2024-09-17T09:29:04.819Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:29:05.163 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 bash[51835]: cd4238c9061fbd0fe2850bd03af1e6a84d2000bcc23dddbbe8334dc3b97d9296 2024-09-17T09:29:05.163 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:29:05.163 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:04 smithi103 podman[51948]: 2024-09-17 09:29:04.876322467 +0000 UTC m=+0.049741922 container died cd4238c9061fbd0fe2850bd03af1e6a84d2000bcc23dddbbe8334dc3b97d9296 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:29:05.460 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:05 smithi103 podman[51948]: 2024-09-17 09:29:05.459180255 +0000 UTC m=+0.632599710 container remove cd4238c9061fbd0fe2850bd03af1e6a84d2000bcc23dddbbe8334dc3b97d9296 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:29:05.460 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:05 smithi103 ceph-mon[49967]: pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:05.460 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:05 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:05.460 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:05 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:05.461 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:05 smithi103 ceph-mon[49967]: Reconfiguring osd.6 (monmap changed)... 2024-09-17T09:29:05.461 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:05 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T09:29:05.461 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:05 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:05.461 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:05 smithi103 ceph-mon[49967]: Reconfiguring daemon osd.6 on smithi103 2024-09-17T09:29:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[60489]: pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[60489]: Reconfiguring osd.6 (monmap changed)... 2024-09-17T09:29:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T09:29:05.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:05.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[60489]: Reconfiguring daemon osd.6 on smithi103 2024-09-17T09:29:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[58326]: pgmap v39: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:05.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[58326]: Reconfiguring osd.6 (monmap changed)... 2024-09-17T09:29:05.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T09:29:05.552 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:05.552 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:05 smithi029 ceph-mon[58326]: Reconfiguring daemon osd.6 on smithi103 2024-09-17T09:29:05.754 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:05 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:29:05.755 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:05 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:29:06.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:06 smithi029 ceph-mon[58326]: from='client.44340 172.21.15.29:0/2679316000' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:29:06.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:06 smithi029 ceph-mon[58326]: from='client.44346 172.21.15.29:0/2243588963' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:29:06.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:06 smithi029 ceph-mon[60489]: from='client.44340 172.21.15.29:0/2679316000' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:29:06.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:06 smithi029 ceph-mon[60489]: from='client.44346 172.21.15.29:0/2243588963' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:29:06.656 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:06 smithi103 ceph-mon[49967]: from='client.44340 172.21.15.29:0/2679316000' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:29:06.656 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:06 smithi103 ceph-mon[49967]: from='client.44346 172.21.15.29:0/2243588963' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:29:07.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[58326]: pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:07.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:07.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:07.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[58326]: Reconfiguring osd.7 (monmap changed)... 2024-09-17T09:29:07.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T09:29:07.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:07.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[58326]: Reconfiguring daemon osd.7 on smithi103 2024-09-17T09:29:07.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[60489]: pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:07.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:07.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:07.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[60489]: Reconfiguring osd.7 (monmap changed)... 2024-09-17T09:29:07.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T09:29:07.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:07.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:07 smithi029 ceph-mon[60489]: Reconfiguring daemon osd.7 on smithi103 2024-09-17T09:29:07.908 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:07 smithi103 ceph-mon[49967]: pgmap v40: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:07.908 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:07 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:07.908 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:07 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:07.908 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:07 smithi103 ceph-mon[49967]: Reconfiguring osd.7 (monmap changed)... 2024-09-17T09:29:07.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:07 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T09:29:07.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:07 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:07.909 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:07 smithi103 ceph-mon[49967]: Reconfiguring daemon osd.7 on smithi103 2024-09-17T09:29:09.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:09.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:09.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:08 smithi103 ceph-mon[49967]: Reconfiguring rgw.foo.smithi103.xonepa (monmap changed)... 2024-09-17T09:29:09.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi103.xonepa", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:29:09.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:09.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:08 smithi103 ceph-mon[49967]: Reconfiguring daemon rgw.foo.smithi103.xonepa on smithi103 2024-09-17T09:29:09.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:09.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:09.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:08 smithi029 ceph-mon[58326]: Reconfiguring rgw.foo.smithi103.xonepa (monmap changed)... 2024-09-17T09:29:09.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi103.xonepa", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:29:09.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:09.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:08 smithi029 ceph-mon[58326]: Reconfiguring daemon rgw.foo.smithi103.xonepa on smithi103 2024-09-17T09:29:09.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:09.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:09.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:08 smithi029 ceph-mon[60489]: Reconfiguring rgw.foo.smithi103.xonepa (monmap changed)... 2024-09-17T09:29:09.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi103.xonepa", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:29:09.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:09.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:08 smithi029 ceph-mon[60489]: Reconfiguring daemon rgw.foo.smithi103.xonepa on smithi103 2024-09-17T09:29:10.173 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:29:10.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:09 smithi103 ceph-mon[49967]: pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:10.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:09 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:10.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:09 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:10.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:09 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi103.gwwbkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:29:10.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:09 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:10.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:09 smithi029 ceph-mon[60489]: pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:10.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:09 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:10.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:09 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:10.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:09 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi103.gwwbkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:29:10.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:09 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:10.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:09 smithi029 ceph-mon[58326]: pgmap v41: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:10.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:09 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:10.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:09 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:10.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:09 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi103.gwwbkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:29:10.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:09 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:10.831 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:29:10.832 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (2m) 50s ago 7m 15.0M - 0.20.0 0881eb8f169f 7bd158a6d65c 2024-09-17T09:29:10.832 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (2m) 31s ago 7m 30.5M - 6.7.4 557c83e11646 e361816f7c14 2024-09-17T09:29:10.832 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 unknown 50s ago 6m - - 2024-09-17T09:29:10.832 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (93s) 31s ago 11m 436M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:29:10.832 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (2m) 50s ago 12m 474M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:29:10.832 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (78s) 50s ago 12m 40.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:29:10.832 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (41s) 31s ago 11m 24.3M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 da050285f124 2024-09-17T09:29:10.832 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (56s) 50s ago 11m 25.3M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 0d3d1a65e2c9 2024-09-17T09:29:10.832 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (2m) 50s ago 8m 15.0M - 0.18.1 e5a616e4b9cf 34bfa263169f 2024-09-17T09:29:10.832 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (2m) 31s ago 7m 8866k - 0.18.1 e5a616e4b9cf 1adf1b5baad1 2024-09-17T09:29:10.832 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (11m) 50s ago 11m 48.3M 4096M 16.2.0 24ecd6d5f14c 991731ce08b5 2024-09-17T09:29:10.832 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (10m) 50s ago 10m 46.2M 4096M 16.2.0 24ecd6d5f14c d5bce63fadc8 2024-09-17T09:29:10.833 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (10m) 50s ago 10m 43.8M 4096M 16.2.0 24ecd6d5f14c e9678d2f1685 2024-09-17T09:29:10.833 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (10m) 50s ago 10m 49.8M 4096M 16.2.0 24ecd6d5f14c dcf7122a7d99 2024-09-17T09:29:10.833 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (9m) 31s ago 9m 49.1M 4096M 16.2.0 24ecd6d5f14c 048c13025d32 2024-09-17T09:29:10.833 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (9m) 31s ago 9m 48.0M 4096M 16.2.0 24ecd6d5f14c 10a64919fbc9 2024-09-17T09:29:10.833 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (9m) 31s ago 9m 46.7M 4096M 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:29:10.833 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (8m) 31s ago 8m 52.8M 4096M 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:29:10.833 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 unknown 31s ago 8m - - 2024-09-17T09:29:10.833 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (6m) 50s ago 6m 40.6M - 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:29:10.833 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (6m) 31s ago 6m 40.7M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:29:10.833 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (6m) 50s ago 6m 38.7M - 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:29:10.833 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (6m) 31s ago 6m 38.0M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:29:11.249 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:29:11.249 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:29:11.249 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:29:11.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:29:11.250 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:29:11.250 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:29:11.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:29:11.250 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:29:11.250 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8 2024-09-17T09:29:11.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:29:11.250 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:29:11.250 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:29:11.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:29:11.250 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:29:11.250 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 12, 2024-09-17T09:29:11.250 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 5 2024-09-17T09:29:11.250 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:29:11.251 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:29:11.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: Reconfiguring rgw.smpl.smithi103.gwwbkl (monmap changed)... 2024-09-17T09:29:11.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: Reconfiguring daemon rgw.smpl.smithi103.gwwbkl on smithi103 2024-09-17T09:29:11.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:11.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:11.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:11.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:29:11.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:11.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:11.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:11.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-17T09:29:11.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-17T09:29:11.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-17T09:29:11.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-17T09:29:11.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-17T09:29:11.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-17T09:29:11.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:11.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:11.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T09:29:11.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: Reconfiguring rgw.smpl.smithi103.gwwbkl (monmap changed)... 2024-09-17T09:29:11.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: Reconfiguring daemon rgw.smpl.smithi103.gwwbkl on smithi103 2024-09-17T09:29:11.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:11.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:11.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T09:29:11.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: Reconfiguring rgw.smpl.smithi103.gwwbkl (monmap changed)... 2024-09-17T09:29:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: Reconfiguring daemon rgw.smpl.smithi103.gwwbkl on smithi103 2024-09-17T09:29:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:29:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-17T09:29:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-17T09:29:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-17T09:29:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-17T09:29:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-17T09:29:11.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-17T09:29:11.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:11.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:11.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T09:29:11.594 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:29:11.594 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:29:11.595 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:29:11.595 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:29:11.595 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [ 2024-09-17T09:29:11.595 INFO:teuthology.orchestra.run.smithi029.stdout: "mon", 2024-09-17T09:29:11.595 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr" 2024-09-17T09:29:11.595 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:29:11.595 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "5/23 daemons upgraded", 2024-09-17T09:29:11.595 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T09:29:11.595 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:29:11.595 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:29:12.007 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:29:12.007 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:29:12.007 INFO:teuthology.orchestra.run.smithi029.stdout: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:29:12.007 INFO:teuthology.orchestra.run.smithi029.stdout: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:29:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:12 smithi103 ceph-mon[49967]: pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:12 smithi103 ceph-mon[49967]: from='client.54241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:12 smithi103 ceph-mon[49967]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:12 smithi103 ceph-mon[49967]: Upgrade: Setting container_image for all mon 2024-09-17T09:29:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:12 smithi103 ceph-mon[49967]: Upgrade: Setting container_image for all crash 2024-09-17T09:29:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:12 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T09:29:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:12 smithi103 ceph-mon[49967]: Upgrade: osd.0 is safe to restart 2024-09-17T09:29:12.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:12 smithi103 ceph-mon[49967]: from='client.54253 172.21.15.29:0/2729868823' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[58326]: pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[58326]: from='client.54241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[58326]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[58326]: Upgrade: Setting container_image for all mon 2024-09-17T09:29:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[58326]: Upgrade: Setting container_image for all crash 2024-09-17T09:29:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T09:29:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[58326]: Upgrade: osd.0 is safe to restart 2024-09-17T09:29:12.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[58326]: from='client.54253 172.21.15.29:0/2729868823' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[60489]: pgmap v42: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[60489]: from='client.54241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[60489]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[60489]: Upgrade: Setting container_image for all mon 2024-09-17T09:29:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[60489]: Upgrade: Setting container_image for all crash 2024-09-17T09:29:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T09:29:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[60489]: Upgrade: osd.0 is safe to restart 2024-09-17T09:29:12.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:12 smithi029 ceph-mon[60489]: from='client.54253 172.21.15.29:0/2729868823' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:13.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:13 smithi029 ceph-mon[58326]: from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:13.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:13 smithi029 ceph-mon[58326]: from='client.44382 172.21.15.29:0/3721221436' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:29:13.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:13.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T09:29:13.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:13.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:13 smithi029 ceph-mon[60489]: from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:13.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:13 smithi029 ceph-mon[60489]: from='client.44382 172.21.15.29:0/3721221436' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:29:13.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:13.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T09:29:13.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:13.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:13 smithi103 ceph-mon[49967]: from='client.44376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:13.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:13 smithi103 ceph-mon[49967]: from='client.44382 172.21.15.29:0/3721221436' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:29:13.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:13.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T09:29:13.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:14.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:14 smithi029 ceph-mon[58326]: pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:14.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:14 smithi029 ceph-mon[58326]: Upgrade: Updating osd.0 2024-09-17T09:29:14.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:14 smithi029 ceph-mon[58326]: Deploying daemon osd.0 on smithi029 2024-09-17T09:29:14.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:14 smithi029 ceph-mon[60489]: pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:14.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:14 smithi029 ceph-mon[60489]: Upgrade: Updating osd.0 2024-09-17T09:29:14.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:14 smithi029 ceph-mon[60489]: Deploying daemon osd.0 on smithi029 2024-09-17T09:29:14.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:14 smithi103 ceph-mon[49967]: pgmap v43: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:14.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:14 smithi103 ceph-mon[49967]: Upgrade: Updating osd.0 2024-09-17T09:29:14.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:14 smithi103 ceph-mon[49967]: Deploying daemon osd.0 on smithi029 2024-09-17T09:29:15.467 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:15 smithi029 systemd[1]: Stopping Ceph osd.0 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:29:15.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.0[29244]: debug 2024-09-17T09:29:15.621+0000 7fbb8f492700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T09:29:15.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.0[29244]: debug 2024-09-17T09:29:15.621+0000 7fbb8f492700 -1 osd.0 78 *** Got signal Terminated *** 2024-09-17T09:29:15.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:15 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.0[29244]: debug 2024-09-17T09:29:15.621+0000 7fbb8f492700 -1 osd.0 78 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:29:15.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:15 smithi029 podman[64364]: 2024-09-17 09:29:15.662146469 +0000 UTC m=+0.155533367 container died 991731ce08b591df3eb8e741e5d1cfc5012dad7b2ff489431f7e30cdbc13e0fd (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-09-17T09:29:15.847 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:15 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 9. 2024-09-17T09:29:15.847 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:15 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:29:15.847 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:15 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:29:15.848 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:15 smithi103 podman[52408]: 2024-09-17 09:29:15.600291443 +0000 UTC m=+0.018101083 image pull de242295e2257c37c8cadfd962369228f8f10b2d48a44259b65fef44ad4f6490 docker.io/prom/prometheus:v2.18.1 2024-09-17T09:29:15.848 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:15 smithi103 podman[52408]: 2024-09-17 09:29:15.722763171 +0000 UTC m=+0.140572803 container create 702ddf784e3722353fa29b6ee0295134f0a64f8a066c2216e34d8be04029460c (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:29:16.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:16.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.7 2024-09-17T09:29:16.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 failed (root=default,host=smithi029) (connection refused reported by osd.7) 2024-09-17T09:29:16.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.2 2024-09-17T09:29:16.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.3 2024-09-17T09:29:16.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.5 2024-09-17T09:29:16.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.1 2024-09-17T09:29:16.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.6 2024-09-17T09:29:16.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.4 2024-09-17T09:29:16.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.7 2024-09-17T09:29:16.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.2 2024-09-17T09:29:16.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.3 2024-09-17T09:29:16.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.5 2024-09-17T09:29:16.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.1 2024-09-17T09:29:16.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.6 2024-09-17T09:29:16.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-mon[49967]: osd.0 reported immediately failed by osd.4 2024-09-17T09:29:16.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:15 smithi103 podman[52408]: 2024-09-17 09:29:15.982951385 +0000 UTC m=+0.400761014 container init 702ddf784e3722353fa29b6ee0295134f0a64f8a066c2216e34d8be04029460c (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:29:16.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:15 smithi103 podman[52408]: 2024-09-17 09:29:15.985866111 +0000 UTC m=+0.403675756 container start 702ddf784e3722353fa29b6ee0295134f0a64f8a066c2216e34d8be04029460c (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:29:16.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.016Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T09:29:16.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.016Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2024-09-17T09:29:16.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.017Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2024-09-17T09:29:16.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.017Z caller=main.go:339 host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:29:16.258 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.017Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.017Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.021Z caller=main.go:678 msg="Starting TSDB ..." 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.021Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.026Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.026Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=16 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.038Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=16 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.038Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=16 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.038Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=16 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.039Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=16 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.039Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=16 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.039Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=6 maxSegment=16 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.039Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=7 maxSegment=16 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.039Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=8 maxSegment=16 2024-09-17T09:29:16.259 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.039Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=9 maxSegment=16 2024-09-17T09:29:16.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.040Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=10 maxSegment=16 2024-09-17T09:29:16.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.040Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=11 maxSegment=16 2024-09-17T09:29:16.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.040Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=12 maxSegment=16 2024-09-17T09:29:16.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.040Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=13 maxSegment=16 2024-09-17T09:29:16.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.041Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=14 maxSegment=16 2024-09-17T09:29:16.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.041Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=15 maxSegment=16 2024-09-17T09:29:16.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.041Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=16 maxSegment=16 2024-09-17T09:29:16.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.042Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=15.907466ms 2024-09-17T09:29:16.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.044Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2024-09-17T09:29:16.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.044Z caller=main.go:695 msg="TSDB started" 2024-09-17T09:29:16.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.044Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:29:16.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.044Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2024-09-17T09:29:16.260 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.044Z caller=main.go:561 msg="Stopping notify discovery manager..." 2024-09-17T09:29:16.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.044Z caller=main.go:583 msg="Stopping scrape manager..." 2024-09-17T09:29:16.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.044Z caller=main.go:557 msg="Notify discovery manager stopped" 2024-09-17T09:29:16.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.044Z caller=main.go:543 msg="Scrape discovery manager stopped" 2024-09-17T09:29:16.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.044Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:29:16.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.044Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2024-09-17T09:29:16.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.044Z caller=main.go:577 msg="Scrape manager stopped" 2024-09-17T09:29:16.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.044Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2024-09-17T09:29:16.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=info ts=2024-09-17T09:29:16.044Z caller=main.go:749 msg="Notifier manager stopped" 2024-09-17T09:29:16.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a[52423]: level=error ts=2024-09-17T09:29:16.044Z caller=main.go:758 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 14: field http_sd_configs not found in type config.plain\n line 27: field http_sd_configs not found in type config.plain\n line 31: field http_sd_configs not found in type config.plain\n line 45: field http_sd_configs not found in type config.plain\n line 49: field http_sd_configs not found in type config.plain" 2024-09-17T09:29:16.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 bash[52408]: 702ddf784e3722353fa29b6ee0295134f0a64f8a066c2216e34d8be04029460c 2024-09-17T09:29:16.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:29:16.261 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 podman[52441]: 2024-09-17 09:29:16.076216363 +0000 UTC m=+0.024511215 container died 702ddf784e3722353fa29b6ee0295134f0a64f8a066c2216e34d8be04029460c (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:29:16.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:16 smithi029 podman[64364]: 2024-09-17 09:29:16.032835208 +0000 UTC m=+0.526222109 container cleanup 991731ce08b591df3eb8e741e5d1cfc5012dad7b2ff489431f7e30cdbc13e0fd (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.0, org.label-schema.vendor=CentOS, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, maintainer=Dimitri Savineau , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-16.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-09-17T09:29:16.278 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:16 smithi029 bash[64364]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.0 2024-09-17T09:29:16.278 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:16.278 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.7 2024-09-17T09:29:16.278 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 failed (root=default,host=smithi029) (connection refused reported by osd.7) 2024-09-17T09:29:16.278 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.2 2024-09-17T09:29:16.278 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.3 2024-09-17T09:29:16.278 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.5 2024-09-17T09:29:16.278 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.1 2024-09-17T09:29:16.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.6 2024-09-17T09:29:16.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.4 2024-09-17T09:29:16.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.7 2024-09-17T09:29:16.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.2 2024-09-17T09:29:16.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.3 2024-09-17T09:29:16.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.5 2024-09-17T09:29:16.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.1 2024-09-17T09:29:16.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.6 2024-09-17T09:29:16.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[60489]: osd.0 reported immediately failed by osd.4 2024-09-17T09:29:16.279 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: pgmap v44: 161 pgs: 161 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:16.279 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.7 2024-09-17T09:29:16.279 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 failed (root=default,host=smithi029) (connection refused reported by osd.7) 2024-09-17T09:29:16.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.2 2024-09-17T09:29:16.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.3 2024-09-17T09:29:16.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.5 2024-09-17T09:29:16.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.1 2024-09-17T09:29:16.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.6 2024-09-17T09:29:16.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.4 2024-09-17T09:29:16.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.7 2024-09-17T09:29:16.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.2 2024-09-17T09:29:16.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.3 2024-09-17T09:29:16.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.5 2024-09-17T09:29:16.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.1 2024-09-17T09:29:16.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.6 2024-09-17T09:29:16.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:16 smithi029 ceph-mon[58326]: osd.0 reported immediately failed by osd.4 2024-09-17T09:29:16.547 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:16 smithi029 podman[64376]: 2024-09-17 09:29:16.277192573 +0000 UTC m=+0.614371667 container remove 991731ce08b591df3eb8e741e5d1cfc5012dad7b2ff489431f7e30cdbc13e0fd (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.build-date=20201204) 2024-09-17T09:29:17.009 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 podman[52441]: 2024-09-17 09:29:16.661796731 +0000 UTC m=+0.610091584 container remove 702ddf784e3722353fa29b6ee0295134f0a64f8a066c2216e34d8be04029460c (image=docker.io/prom/prometheus:v2.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus.a, maintainer=The Prometheus Authors ) 2024-09-17T09:29:17.009 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Main process exited, code=exited, status=1/FAILURE 2024-09-17T09:29:17.009 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:16 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:29:17.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:17 smithi029 ceph-mon[58326]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:29:17.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:17 smithi029 ceph-mon[58326]: osdmap e79: 8 total, 7 up, 8 in 2024-09-17T09:29:17.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:17 smithi029 ceph-mon[60489]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:29:17.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:17 smithi029 ceph-mon[60489]: osdmap e79: 8 total, 7 up, 8 in 2024-09-17T09:29:17.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:16 smithi029 podman[64485]: 2024-09-17 09:29:16.762543758 +0000 UTC m=+0.110255693 container create 9b93184495698c13af6b4e33a87e64db25cf246b3fffd108623e9bc1e9cc9cf1 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-17T09:29:17.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:16 smithi029 podman[64485]: 2024-09-17 09:29:16.668853363 +0000 UTC m=+0.016565269 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:29:17.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:17 smithi103 ceph-mon[49967]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:29:17.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:17 smithi103 ceph-mon[49967]: osdmap e79: 8 total, 7 up, 8 in 2024-09-17T09:29:17.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:17 smithi029 podman[64485]: 2024-09-17 09:29:17.051979909 +0000 UTC m=+0.399691812 container init 9b93184495698c13af6b4e33a87e64db25cf246b3fffd108623e9bc1e9cc9cf1 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:29:17.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:17 smithi029 podman[64485]: 2024-09-17 09:29:17.055747648 +0000 UTC m=+0.403459550 container start 9b93184495698c13af6b4e33a87e64db25cf246b3fffd108623e9bc1e9cc9cf1 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-17T09:29:17.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:17 smithi029 podman[64485]: 2024-09-17 09:29:17.135768702 +0000 UTC m=+0.483480602 container attach 9b93184495698c13af6b4e33a87e64db25cf246b3fffd108623e9bc1e9cc9cf1 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-09-17T09:29:17.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:17 smithi029 podman[64485]: 2024-09-17 09:29:17.256463401 +0000 UTC m=+0.604175295 container died 9b93184495698c13af6b4e33a87e64db25cf246b3fffd108623e9bc1e9cc9cf1 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-17T09:29:18.228 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:18 smithi029 ceph-mon[60489]: pgmap v46: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T09:29:18.229 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:18 smithi029 ceph-mon[60489]: osdmap e80: 8 total, 7 up, 8 in 2024-09-17T09:29:18.229 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:18 smithi029 ceph-mon[58326]: pgmap v46: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T09:29:18.229 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:18 smithi029 ceph-mon[58326]: osdmap e80: 8 total, 7 up, 8 in 2024-09-17T09:29:18.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:17 smithi029 podman[64509]: 2024-09-17 09:29:17.96948677 +0000 UTC m=+0.707096571 container remove 9b93184495698c13af6b4e33a87e64db25cf246b3fffd108623e9bc1e9cc9cf1 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-17T09:29:18.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:17 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.0.service: Deactivated successfully. 2024-09-17T09:29:18.229 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:17 smithi029 systemd[1]: Stopped Ceph osd.0 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:29:18.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:17 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.0.service: Consumed 1.147s CPU time. 2024-09-17T09:29:18.230 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:18 smithi029 systemd[1]: Starting Ceph osd.0 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:29:18.463 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:18 smithi103 ceph-mon[49967]: pgmap v46: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T09:29:18.463 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:18 smithi103 ceph-mon[49967]: osdmap e80: 8 total, 7 up, 8 in 2024-09-17T09:29:19.502 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 podman[64693]: 2024-09-17 09:29:19.169187211 +0000 UTC m=+0.196423790 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:29:19.503 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 podman[64693]: 2024-09-17 09:29:19.272102485 +0000 UTC m=+0.299339049 container create a0d6e670d43d8e5f51110083d36502e709e1aae22b1cb1facd12670aa437faa5 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-activate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-17T09:29:19.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 podman[64693]: 2024-09-17 09:29:19.501855878 +0000 UTC m=+0.529092452 container init a0d6e670d43d8e5f51110083d36502e709e1aae22b1cb1facd12670aa437faa5 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-17T09:29:19.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 podman[64693]: 2024-09-17 09:29:19.505499901 +0000 UTC m=+0.532736475 container start a0d6e670d43d8e5f51110083d36502e709e1aae22b1cb1facd12670aa437faa5 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:29:19.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 podman[64693]: 2024-09-17 09:29:19.585659889 +0000 UTC m=+0.612896469 container attach a0d6e670d43d8e5f51110083d36502e709e1aae22b1cb1facd12670aa437faa5 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-activate, GIT_CLEAN=True, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:29:20.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:20 smithi029 ceph-mon[58326]: pgmap v48: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T09:29:20.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:20 smithi029 ceph-mon[58326]: from='client.54283 172.21.15.29:0/2603257063' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:29:20.067 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:20 smithi029 ceph-mon[60489]: pgmap v48: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T09:29:20.067 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:20 smithi029 ceph-mon[60489]: from='client.54283 172.21.15.29:0/2603257063' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:29:20.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-activate[64746]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T09:29:20.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 bash[64693]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T09:29:20.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-activate[64746]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T09:29:20.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 bash[64693]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T09:29:20.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-activate[64746]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T09:29:20.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 bash[64693]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T09:29:20.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-activate[64746]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T09:29:20.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 bash[64693]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T09:29:20.068 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-activate[64746]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-17T09:29:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 bash[64693]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-17T09:29:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-activate[64746]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T09:29:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 bash[64693]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T09:29:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-activate[64746]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-17T09:29:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 bash[64693]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-17T09:29:20.069 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:19 smithi029 podman[64693]: 2024-09-17 09:29:19.864065079 +0000 UTC m=+0.891301644 container died a0d6e670d43d8e5f51110083d36502e709e1aae22b1cb1facd12670aa437faa5 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:29:20.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:20 smithi103 ceph-mon[49967]: pgmap v48: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T09:29:20.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:20 smithi103 ceph-mon[49967]: from='client.54283 172.21.15.29:0/2603257063' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:29:20.798 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:20 smithi029 podman[64912]: 2024-09-17 09:29:20.453342042 +0000 UTC m=+0.579603446 container remove a0d6e670d43d8e5f51110083d36502e709e1aae22b1cb1facd12670aa437faa5 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-activate, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-17T09:29:21.129 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:21 smithi029 ceph-mon[58326]: from='client.54289 172.21.15.29:0/2134246776' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:29:21.129 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:21 smithi029 ceph-mon[60489]: from='client.54289 172.21.15.29:0/2134246776' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:29:21.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:21 smithi103 ceph-mon[49967]: from='client.54289 172.21.15.29:0/2134246776' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:29:22.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:22 smithi029 ceph-mon[58326]: pgmap v49: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:22.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:22 smithi029 ceph-mon[60489]: pgmap v49: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:22.135 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:22 smithi029 podman[65000]: 2024-09-17 09:29:21.928988954 +0000 UTC m=+0.748438490 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:29:22.135 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:22 smithi029 podman[65000]: 2024-09-17 09:29:22.031364192 +0000 UTC m=+0.850813723 container create 4c456435000b33b3310d9b3cfc32c3eb25af1f800cb692b773dc24f8555c4924 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b) 2024-09-17T09:29:22.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:22 smithi103 ceph-mon[49967]: pgmap v49: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:22.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:22 smithi029 podman[65000]: 2024-09-17 09:29:22.262692522 +0000 UTC m=+1.082142042 container init 4c456435000b33b3310d9b3cfc32c3eb25af1f800cb692b773dc24f8555c4924 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-17T09:29:22.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:22 smithi029 podman[65000]: 2024-09-17 09:29:22.266444687 +0000 UTC m=+1.085894208 container start 4c456435000b33b3310d9b3cfc32c3eb25af1f800cb692b773dc24f8555c4924 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:29:22.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:22 smithi029 ceph-osd[65037]: -- 172.21.15.29:0/2099809161 <== mon.1 v2:172.21.15.29:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b2585ff1e0 con 0x55b258772400 2024-09-17T09:29:22.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:22 smithi029 bash[65000]: 4c456435000b33b3310d9b3cfc32c3eb25af1f800cb692b773dc24f8555c4924 2024-09-17T09:29:22.548 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:22 smithi029 systemd[1]: Started Ceph osd.0 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:29:23.102 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0[65033]: 2024-09-17T09:29:22.805+0000 7fe7a885d640 -1 Falling back to public interface 2024-09-17T09:29:23.753 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:23 smithi029 ceph-mon[60489]: pgmap v50: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:23.753 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:23 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:23.753 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:23 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:23.753 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:23 smithi029 ceph-mon[60489]: Health check failed: Degraded data redundancy: 115/999 objects degraded (11.512%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T09:29:23.754 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:23 smithi029 ceph-mon[58326]: pgmap v50: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:23.754 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:23 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:23.754 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:23 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:23.754 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:23 smithi029 ceph-mon[58326]: Health check failed: Degraded data redundancy: 115/999 objects degraded (11.512%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T09:29:23.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:23 smithi103 ceph-mon[49967]: pgmap v50: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:23.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:23 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:23.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:23 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:23.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:23 smithi103 ceph-mon[49967]: Health check failed: Degraded data redundancy: 115/999 objects degraded (11.512%), 19 pgs degraded (PG_DEGRADED) 2024-09-17T09:29:25.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:25 smithi103 ceph-mon[49967]: pgmap v51: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:25.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:25 smithi029 ceph-mon[58326]: pgmap v51: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:25.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:25 smithi029 ceph-mon[60489]: pgmap v51: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:25.799 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0[65033]: 2024-09-17T09:29:25.518+0000 7fe7a885d640 -1 osd.0 0 read_superblock omap replica is missing. 2024-09-17T09:29:25.799 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:25 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0[65033]: 2024-09-17T09:29:25.542+0000 7fe7a885d640 -1 osd.0 78 log_to_monitors true 2024-09-17T09:29:26.718 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:26 smithi029 ceph-mon[60489]: from='osd.0 [v2:172.21.15.29:6802/2014973713,v1:172.21.15.29:6803/2014973713]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:29:26.719 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:26 smithi029 ceph-mon[60489]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:29:26.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:26 smithi029 ceph-mon[58326]: from='osd.0 [v2:172.21.15.29:6802/2014973713,v1:172.21.15.29:6803/2014973713]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:29:26.719 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:26 smithi029 ceph-mon[58326]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:29:26.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:26 smithi103 ceph-mon[49967]: from='osd.0 [v2:172.21.15.29:6802/2014973713,v1:172.21.15.29:6803/2014973713]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:29:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:26 smithi103 ceph-mon[49967]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T09:29:27.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:29:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0[65033]: 2024-09-17T09:29:26.717+0000 7fe79c5ff640 -1 osd.0 78 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:29:27.255 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:26 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Scheduled restart job, restart counter is at 10. 2024-09-17T09:29:27.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:26 smithi103 systemd[1]: Stopped Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:29:27.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:26 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Start request repeated too quickly. 2024-09-17T09:29:27.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:26 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service: Failed with result 'exit-code'. 2024-09-17T09:29:27.256 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:29:26 smithi103 systemd[1]: Failed to start Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:29:27.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:27 smithi103 ceph-mon[49967]: pgmap v52: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:27.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:27 smithi103 ceph-mon[49967]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T09:29:27.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:27 smithi103 ceph-mon[49967]: osdmap e81: 8 total, 7 up, 8 in 2024-09-17T09:29:27.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:27 smithi103 ceph-mon[49967]: from='osd.0 [v2:172.21.15.29:6802/2014973713,v1:172.21.15.29:6803/2014973713]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:29:27.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:27 smithi103 ceph-mon[49967]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:29:27.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:27 smithi103 ceph-mon[49967]: from='osd.0 ' entity='osd.0' 2024-09-17T09:29:27.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:27 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:27.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:27 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:27.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[58326]: pgmap v52: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:27.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[58326]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T09:29:27.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[58326]: osdmap e81: 8 total, 7 up, 8 in 2024-09-17T09:29:27.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[58326]: from='osd.0 [v2:172.21.15.29:6802/2014973713,v1:172.21.15.29:6803/2014973713]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:29:27.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[58326]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:29:27.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[58326]: from='osd.0 ' entity='osd.0' 2024-09-17T09:29:27.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:27.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:27.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[60489]: pgmap v52: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:27.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[60489]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T09:29:27.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[60489]: osdmap e81: 8 total, 7 up, 8 in 2024-09-17T09:29:27.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[60489]: from='osd.0 [v2:172.21.15.29:6802/2014973713,v1:172.21.15.29:6803/2014973713]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:29:27.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[60489]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:29:27.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[60489]: from='osd.0 ' entity='osd.0' 2024-09-17T09:29:27.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:27.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:27 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:28.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:28.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:28.990 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:28.990 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:28.990 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:28 smithi029 ceph-mon[60489]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:29:28.990 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:28 smithi029 ceph-mon[60489]: osd.0 [v2:172.21.15.29:6802/2014973713,v1:172.21.15.29:6803/2014973713] boot 2024-09-17T09:29:28.990 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:28 smithi029 ceph-mon[60489]: osdmap e82: 8 total, 8 up, 8 in 2024-09-17T09:29:28.990 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:29:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:28 smithi029 ceph-mon[58326]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:29:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:28 smithi029 ceph-mon[58326]: osd.0 [v2:172.21.15.29:6802/2014973713,v1:172.21.15.29:6803/2014973713] boot 2024-09-17T09:29:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:28 smithi029 ceph-mon[58326]: osdmap e82: 8 total, 8 up, 8 in 2024-09-17T09:29:28.991 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:29:29.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:29.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:29.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:28 smithi103 ceph-mon[49967]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:29:29.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:28 smithi103 ceph-mon[49967]: osd.0 [v2:172.21.15.29:6802/2014973713,v1:172.21.15.29:6803/2014973713] boot 2024-09-17T09:29:29.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:28 smithi103 ceph-mon[49967]: osdmap e82: 8 total, 8 up, 8 in 2024-09-17T09:29:29.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T09:29:30.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:29 smithi103 ceph-mon[49967]: pgmap v55: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:30.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:29 smithi103 ceph-mon[49967]: osdmap e83: 8 total, 8 up, 8 in 2024-09-17T09:29:30.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:29 smithi029 ceph-mon[58326]: pgmap v55: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:30.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:29 smithi029 ceph-mon[58326]: osdmap e83: 8 total, 8 up, 8 in 2024-09-17T09:29:30.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:29 smithi029 ceph-mon[60489]: pgmap v55: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:30.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:29 smithi029 ceph-mon[60489]: osdmap e83: 8 total, 8 up, 8 in 2024-09-17T09:29:31.298 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:29:31.037+0000 7fe73b7e7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T09:29:32.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: pgmap v57: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:32.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:32.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:32.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:32.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:29:32.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:29:32.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: pgmap v58: 161 pgs: 13 active+undersized, 8 active+undersized+degraded, 140 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 53/999 objects degraded (5.305%) 2024-09-17T09:29:32.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: pgmap v59: 161 pgs: 13 active+undersized, 8 active+undersized+degraded, 140 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 53/999 objects degraded (5.305%) 2024-09-17T09:29:32.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:32.121 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:29:32.121 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:32.121 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:32.121 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:32.121 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T09:29:32.121 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T09:29:32.121 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[58326]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T09:29:32.121 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: pgmap v57: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:32.121 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:32.121 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:32.122 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:32.122 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:29:32.122 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:29:32.122 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: pgmap v58: 161 pgs: 13 active+undersized, 8 active+undersized+degraded, 140 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 53/999 objects degraded (5.305%) 2024-09-17T09:29:32.122 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: pgmap v59: 161 pgs: 13 active+undersized, 8 active+undersized+degraded, 140 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 53/999 objects degraded (5.305%) 2024-09-17T09:29:32.122 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:32.122 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:29:32.122 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:32.122 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:32.122 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:32.122 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T09:29:32.122 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T09:29:32.122 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:31 smithi029 ceph-mon[60489]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T09:29:32.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: pgmap v57: 161 pgs: 39 active+undersized, 19 active+undersized+degraded, 103 active+clean; 604 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 115/999 objects degraded (11.512%) 2024-09-17T09:29:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:29:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:29:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: pgmap v58: 161 pgs: 13 active+undersized, 8 active+undersized+degraded, 140 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 53/999 objects degraded (5.305%) 2024-09-17T09:29:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: pgmap v59: 161 pgs: 13 active+undersized, 8 active+undersized+degraded, 140 active+clean; 604 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 53/999 objects degraded (5.305%) 2024-09-17T09:29:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:29:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:32.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:32.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T09:29:32.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T09:29:32.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:31 smithi103 ceph-mon[49967]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T09:29:33.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:32 smithi029 ceph-mon[58326]: Health check update: Degraded data redundancy: 53/999 objects degraded (5.305%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T09:29:33.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:32 smithi029 ceph-mon[60489]: Health check update: Degraded data redundancy: 53/999 objects degraded (5.305%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T09:29:33.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:32 smithi103 ceph-mon[49967]: Health check update: Degraded data redundancy: 53/999 objects degraded (5.305%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T09:29:34.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:33 smithi029 ceph-mon[60489]: pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:34.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:33 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/1575606013' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:29:34.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:33 smithi029 ceph-mon[60489]: from='client.54319 172.21.15.29:0/2108051251' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:29:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:33 smithi029 ceph-mon[58326]: pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:33 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/1575606013' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:29:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:33 smithi029 ceph-mon[58326]: from='client.54319 172.21.15.29:0/2108051251' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:29:34.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:33 smithi103 ceph-mon[49967]: pgmap v60: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:34.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:33 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/1575606013' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:29:34.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:33 smithi103 ceph-mon[49967]: from='client.54319 172.21.15.29:0/2108051251' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:29:35.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:34 smithi103 ceph-mon[49967]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/999 objects degraded (5.305%), 8 pgs degraded) 2024-09-17T09:29:35.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:34 smithi029 ceph-mon[58326]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/999 objects degraded (5.305%), 8 pgs degraded) 2024-09-17T09:29:35.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:34 smithi029 ceph-mon[60489]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/999 objects degraded (5.305%), 8 pgs degraded) 2024-09-17T09:29:36.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:35 smithi103 ceph-mon[49967]: pgmap v61: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:36.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:35 smithi029 ceph-mon[58326]: pgmap v61: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:35 smithi029 ceph-mon[60489]: pgmap v61: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:38.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:37 smithi103 ceph-mon[49967]: pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 125 B/s rd, 0 op/s 2024-09-17T09:29:38.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:37 smithi029 ceph-mon[58326]: pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 125 B/s rd, 0 op/s 2024-09-17T09:29:38.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:37 smithi029 ceph-mon[60489]: pgmap v62: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 125 B/s rd, 0 op/s 2024-09-17T09:29:40.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:39 smithi103 ceph-mon[49967]: pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 118 B/s rd, 0 op/s 2024-09-17T09:29:40.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:39 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:40.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:39 smithi029 ceph-mon[58326]: pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 118 B/s rd, 0 op/s 2024-09-17T09:29:40.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:39 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:40.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:39 smithi029 ceph-mon[60489]: pgmap v63: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 118 B/s rd, 0 op/s 2024-09-17T09:29:40.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:39 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:42.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:41 smithi103 ceph-mon[49967]: pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T09:29:42.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:41 smithi029 ceph-mon[58326]: pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T09:29:42.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:41 smithi029 ceph-mon[60489]: pgmap v64: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T09:29:42.374 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:29:43.032 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:29:43.032 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (3m) 15s ago 7m 19.1M - 0.20.0 0881eb8f169f 7bd158a6d65c 2024-09-17T09:29:43.032 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (2m) 63s ago 7m 30.5M - 6.7.4 557c83e11646 e361816f7c14 2024-09-17T09:29:43.032 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 unknown 15s ago 6m - - 2024-09-17T09:29:43.032 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (2m) 63s ago 11m 436M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:29:43.032 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (2m) 15s ago 13m 481M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:29:43.032 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (110s) 15s ago 13m 45.1M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:29:43.032 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (73s) 63s ago 12m 24.3M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 da050285f124 2024-09-17T09:29:43.032 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (88s) 15s ago 12m 40.9M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 0d3d1a65e2c9 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (3m) 15s ago 8m 15.0M - 0.18.1 e5a616e4b9cf 34bfa263169f 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (3m) 63s ago 8m 8866k - 0.18.1 e5a616e4b9cf 1adf1b5baad1 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (21s) 15s ago 11m 12.1M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 4c456435000b 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (11m) 15s ago 11m 47.0M 4096M 16.2.0 24ecd6d5f14c d5bce63fadc8 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (10m) 15s ago 10m 44.7M 4096M 16.2.0 24ecd6d5f14c e9678d2f1685 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (10m) 15s ago 10m 50.8M 4096M 16.2.0 24ecd6d5f14c dcf7122a7d99 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (10m) 63s ago 10m 49.1M 4096M 16.2.0 24ecd6d5f14c 048c13025d32 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (10m) 63s ago 9m 48.0M 4096M 16.2.0 24ecd6d5f14c 10a64919fbc9 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (9m) 63s ago 9m 46.7M 4096M 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (9m) 63s ago 9m 52.8M 4096M 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 unknown 63s ago 8m - - 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (7m) 15s ago 7m 41.2M - 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (7m) 63s ago 7m 40.7M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:29:43.033 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (7m) 15s ago 7m 38.9M - 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:29:43.034 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (7m) 63s ago 7m 38.0M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:29:43.435 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:29:43.435 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:29:43.435 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:29:43.435 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:29:43.435 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 7, 2024-09-17T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 11, 2024-09-17T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 6 2024-09-17T09:29:43.436 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:29:43.437 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:29:43.772 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:29:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:29:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:29:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:29:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [ 2024-09-17T09:29:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mon", 2024-09-17T09:29:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr" 2024-09-17T09:29:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:29:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "6/23 daemons upgraded", 2024-09-17T09:29:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T09:29:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:29:43.773 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:29:44.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:43 smithi029 ceph-mon[60489]: from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:44.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:43 smithi029 ceph-mon[60489]: from='client.44448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:44.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:43 smithi029 ceph-mon[60489]: pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:44.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:43 smithi029 ceph-mon[60489]: from='client.54337 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:44.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:43 smithi029 ceph-mon[60489]: from='client.54343 172.21.15.29:0/1085609948' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:44.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:43 smithi029 ceph-mon[58326]: from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:44.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:43 smithi029 ceph-mon[58326]: from='client.44448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:44.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:43 smithi029 ceph-mon[58326]: pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:44.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:43 smithi029 ceph-mon[58326]: from='client.54337 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:44.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:43 smithi029 ceph-mon[58326]: from='client.54343 172.21.15.29:0/1085609948' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:44.171 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:29:44.171 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:29:44.171 INFO:teuthology.orchestra.run.smithi029.stdout: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:29:44.171 INFO:teuthology.orchestra.run.smithi029.stdout: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:29:44.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:43 smithi103 ceph-mon[49967]: from='client.44442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:44.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:43 smithi103 ceph-mon[49967]: from='client.44448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:44.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:43 smithi103 ceph-mon[49967]: pgmap v65: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:44.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:43 smithi103 ceph-mon[49967]: from='client.54337 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:44.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:43 smithi103 ceph-mon[49967]: from='client.54343 172.21.15.29:0/1085609948' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:29:45.129 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:44 smithi029 ceph-mon[58326]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:45.129 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:44 smithi029 ceph-mon[58326]: from='client.44472 172.21.15.29:0/2248771733' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:29:45.129 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:44 smithi029 ceph-mon[60489]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:45.129 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:44 smithi029 ceph-mon[60489]: from='client.44472 172.21.15.29:0/2248771733' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:29:45.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:44 smithi103 ceph-mon[49967]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:29:45.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:44 smithi103 ceph-mon[49967]: from='client.44472 172.21.15.29:0/2248771733' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:29:46.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:45 smithi029 ceph-mon[58326]: pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:46.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:45 smithi029 ceph-mon[60489]: pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:46.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:45 smithi103 ceph-mon[49967]: pgmap v66: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:46.963 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T09:29:46.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:46 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T09:29:46.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:46 smithi029 ceph-mon[58326]: Upgrade: osd.1 is safe to restart 2024-09-17T09:29:46.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:46 smithi029 ceph-mon[58326]: from='client.54373 172.21.15.29:0/1413885533' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:29:46.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:46 smithi029 ceph-mon[58326]: from='client.44496 172.21.15.29:0/3582298051' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:29:46.964 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T09:29:46.964 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:46 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T09:29:46.964 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:46 smithi029 ceph-mon[60489]: Upgrade: osd.1 is safe to restart 2024-09-17T09:29:46.964 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:46 smithi029 ceph-mon[60489]: from='client.54373 172.21.15.29:0/1413885533' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:29:46.964 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:46 smithi029 ceph-mon[60489]: from='client.44496 172.21.15.29:0/3582298051' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:29:47.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T09:29:47.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:46 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T09:29:47.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:46 smithi103 ceph-mon[49967]: Upgrade: osd.1 is safe to restart 2024-09-17T09:29:47.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:46 smithi103 ceph-mon[49967]: from='client.54373 172.21.15.29:0/1413885533' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:29:47.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:46 smithi103 ceph-mon[49967]: from='client.44496 172.21.15.29:0/3582298051' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:29:48.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:47 smithi029 ceph-mon[58326]: pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:48.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:47 smithi029 ceph-mon[60489]: pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:48.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:47 smithi103 ceph-mon[49967]: pgmap v67: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:50.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:49 smithi103 ceph-mon[49967]: Upgrade: Updating osd.1 2024-09-17T09:29:50.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:49 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:50.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:49 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T09:29:50.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:49 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:50.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:49 smithi103 ceph-mon[49967]: Deploying daemon osd.1 on smithi029 2024-09-17T09:29:50.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:49 smithi103 ceph-mon[49967]: pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:50.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:49 smithi029 ceph-mon[58326]: Upgrade: Updating osd.1 2024-09-17T09:29:50.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:49 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:50.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:49 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T09:29:50.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:49 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:50.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:49 smithi029 ceph-mon[58326]: Deploying daemon osd.1 on smithi029 2024-09-17T09:29:50.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:49 smithi029 ceph-mon[58326]: pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:50.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:49 smithi029 ceph-mon[60489]: Upgrade: Updating osd.1 2024-09-17T09:29:50.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:49 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:50.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:49 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T09:29:50.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:49 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:29:50.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:49 smithi029 ceph-mon[60489]: Deploying daemon osd.1 on smithi029 2024-09-17T09:29:50.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:49 smithi029 ceph-mon[60489]: pgmap v68: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:29:52.222 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:51 smithi029 ceph-mon[60489]: pgmap v69: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:52.222 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:51 smithi029 ceph-mon[58326]: pgmap v69: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:52.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:52 smithi029 systemd[1]: Stopping Ceph osd.1 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:29:52.222 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.1[31661]: debug 2024-09-17T09:29:52.178+0000 7f421f946700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T09:29:52.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.1[31661]: debug 2024-09-17T09:29:52.178+0000 7f421f946700 -1 osd.1 83 *** Got signal Terminated *** 2024-09-17T09:29:52.223 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.1[31661]: debug 2024-09-17T09:29:52.178+0000 7f421f946700 -1 osd.1 83 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:29:52.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:51 smithi103 ceph-mon[49967]: pgmap v69: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:52.528 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:52 smithi029 podman[67545]: 2024-09-17 09:29:52.221113988 +0000 UTC m=+0.143651167 container died d5bce63fadc818307659b727a43bb87eae623f153fec768e785585d9f61aeccb (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.1, CEPH_POINT_RELEASE=-16.2.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=HEAD, ceph=True, maintainer=Dimitri Savineau ) 2024-09-17T09:29:52.798 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:52 smithi029 podman[67545]: 2024-09-17 09:29:52.626110184 +0000 UTC m=+0.548647344 container cleanup d5bce63fadc818307659b727a43bb87eae623f153fec768e785585d9f61aeccb (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.1, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, GIT_BRANCH=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, RELEASE=HEAD) 2024-09-17T09:29:52.798 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:52 smithi029 bash[67545]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.1 2024-09-17T09:29:53.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 failed (root=default,host=smithi029) (connection refused reported by osd.3) 2024-09-17T09:29:53.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.259 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.259 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.259 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.259 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.259 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.259 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.259 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.259 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:52 smithi103 ceph-mon[49967]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:52 smithi029 podman[67557]: 2024-09-17 09:29:52.828742463 +0000 UTC m=+0.602524150 container remove d5bce63fadc818307659b727a43bb87eae623f153fec768e785585d9f61aeccb (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557) 2024-09-17T09:29:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 failed (root=default,host=smithi029) (connection refused reported by osd.3) 2024-09-17T09:29:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[60489]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.302 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.302 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 failed (root=default,host=smithi029) (connection refused reported by osd.3) 2024-09-17T09:29:53.302 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.302 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.302 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.302 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.302 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.302 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.302 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.3 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.6 2024-09-17T09:29:53.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.4 2024-09-17T09:29:53.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.7 2024-09-17T09:29:53.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.5 2024-09-17T09:29:53.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.2 2024-09-17T09:29:53.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:52 smithi029 ceph-mon[58326]: osd.1 reported immediately failed by osd.0 2024-09-17T09:29:53.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:53 smithi029 podman[67659]: 2024-09-17 09:29:53.311834916 +0000 UTC m=+0.017411716 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:29:53.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:53 smithi029 podman[67659]: 2024-09-17 09:29:53.449350703 +0000 UTC m=+0.154927492 container create 02839172ad564c5bd632d2d26ecf68b78f29af345c0746b2d678b7da114774c2 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-17T09:29:53.696 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:53 smithi029 podman[67659]: 2024-09-17 09:29:53.695345832 +0000 UTC m=+0.400922631 container init 02839172ad564c5bd632d2d26ecf68b78f29af345c0746b2d678b7da114774c2 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b) 2024-09-17T09:29:53.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:53 smithi029 podman[67659]: 2024-09-17 09:29:53.699173686 +0000 UTC m=+0.404750475 container start 02839172ad564c5bd632d2d26ecf68b78f29af345c0746b2d678b7da114774c2 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS) 2024-09-17T09:29:53.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:53 smithi029 podman[67659]: 2024-09-17 09:29:53.787490848 +0000 UTC m=+0.493067638 container attach 02839172ad564c5bd632d2d26ecf68b78f29af345c0746b2d678b7da114774c2 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-17T09:29:53.977 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:53 smithi029 podman[67659]: 2024-09-17 09:29:53.8900341 +0000 UTC m=+0.595610890 container died 02839172ad564c5bd632d2d26ecf68b78f29af345c0746b2d678b7da114774c2 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b) 2024-09-17T09:29:54.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:53 smithi103 ceph-mon[49967]: pgmap v70: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:54.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:53 smithi103 ceph-mon[49967]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:29:54.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:53 smithi103 ceph-mon[49967]: osdmap e84: 8 total, 7 up, 8 in 2024-09-17T09:29:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:53 smithi029 ceph-mon[58326]: pgmap v70: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:53 smithi029 ceph-mon[58326]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:29:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:53 smithi029 ceph-mon[58326]: osdmap e84: 8 total, 7 up, 8 in 2024-09-17T09:29:54.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:53 smithi029 ceph-mon[60489]: pgmap v70: 161 pgs: 161 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:29:54.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:53 smithi029 ceph-mon[60489]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:29:54.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:53 smithi029 ceph-mon[60489]: osdmap e84: 8 total, 7 up, 8 in 2024-09-17T09:29:55.005 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:54 smithi029 ceph-mon[58326]: osdmap e85: 8 total, 7 up, 8 in 2024-09-17T09:29:55.005 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:54 smithi029 ceph-mon[60489]: osdmap e85: 8 total, 7 up, 8 in 2024-09-17T09:29:55.005 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:54 smithi029 podman[67682]: 2024-09-17 09:29:54.738188228 +0000 UTC m=+0.842329819 container remove 02839172ad564c5bd632d2d26ecf68b78f29af345c0746b2d678b7da114774c2 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-17T09:29:55.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:54 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.1.service: Deactivated successfully. 2024-09-17T09:29:55.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:54 smithi029 systemd[1]: Stopped Ceph osd.1 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:29:55.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:54 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.1.service: Consumed 1.155s CPU time. 2024-09-17T09:29:55.006 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:54 smithi029 systemd[1]: Starting Ceph osd.1 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:29:55.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:54 smithi103 ceph-mon[49967]: osdmap e85: 8 total, 7 up, 8 in 2024-09-17T09:29:55.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 podman[67779]: 2024-09-17 09:29:55.101152123 +0000 UTC m=+0.016527240 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:29:55.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 podman[67779]: 2024-09-17 09:29:55.209474914 +0000 UTC m=+0.124850018 container create 249dca28ef2ceff006b1d4c65e61de75e41098cef86cf5f95c600e1661802a99 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-17T09:29:55.616 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 podman[67779]: 2024-09-17 09:29:55.522999041 +0000 UTC m=+0.438374157 container init 249dca28ef2ceff006b1d4c65e61de75e41098cef86cf5f95c600e1661802a99 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:29:55.616 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 podman[67779]: 2024-09-17 09:29:55.526562236 +0000 UTC m=+0.441937346 container start 249dca28ef2ceff006b1d4c65e61de75e41098cef86cf5f95c600e1661802a99 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:29:55.616 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 podman[67779]: 2024-09-17 09:29:55.615246446 +0000 UTC m=+0.530621559 container attach 249dca28ef2ceff006b1d4c65e61de75e41098cef86cf5f95c600e1661802a99 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:29:55.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-activate[67801]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T09:29:55.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 bash[67779]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T09:29:55.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-activate[67801]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T09:29:55.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 bash[67779]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T09:29:55.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-activate[67801]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T09:29:55.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 bash[67779]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T09:29:55.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-activate[67801]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T09:29:55.867 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 bash[67779]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T09:29:55.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-activate[67801]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-17T09:29:55.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 bash[67779]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-17T09:29:55.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-activate[67801]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T09:29:55.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 bash[67779]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T09:29:55.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-activate[67801]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-17T09:29:55.868 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 bash[67779]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-17T09:29:56.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:55 smithi029 ceph-mon[58326]: pgmap v73: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T09:29:56.249 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:55 smithi029 ceph-mon[60489]: pgmap v73: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T09:29:56.249 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:55 smithi029 podman[67779]: 2024-09-17 09:29:55.885545108 +0000 UTC m=+0.800920219 container died 249dca28ef2ceff006b1d4c65e61de75e41098cef86cf5f95c600e1661802a99 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-activate, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-17T09:29:56.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:55 smithi103 ceph-mon[49967]: pgmap v73: 161 pgs: 21 stale+active+clean, 140 active+clean; 604 KiB data, 118 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T09:29:56.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:56 smithi029 podman[67933]: 2024-09-17 09:29:56.582985446 +0000 UTC m=+0.690728380 container remove 249dca28ef2ceff006b1d4c65e61de75e41098cef86cf5f95c600e1661802a99 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-17T09:29:56.895 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:56 smithi029 podman[67990]: 2024-09-17 09:29:56.784943024 +0000 UTC m=+0.107308532 container create 2ff250ddb1d9df1dee5642100252b078f51d77b65c1b46cae55128b2ad38ae8f (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-17T09:29:56.896 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:56 smithi029 podman[67990]: 2024-09-17 09:29:56.693784637 +0000 UTC m=+0.016150110 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:29:57.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:57 smithi029 ceph-mon[60489]: Health check failed: Degraded data redundancy: 128/999 objects degraded (12.813%), 20 pgs degraded (PG_DEGRADED) 2024-09-17T09:29:57.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:56 smithi029 podman[67990]: 2024-09-17 09:29:56.999039483 +0000 UTC m=+0.321404955 container init 2ff250ddb1d9df1dee5642100252b078f51d77b65c1b46cae55128b2ad38ae8f (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-17T09:29:57.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:57 smithi029 podman[67990]: 2024-09-17 09:29:57.002522564 +0000 UTC m=+0.324888027 container start 2ff250ddb1d9df1dee5642100252b078f51d77b65c1b46cae55128b2ad38ae8f (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True) 2024-09-17T09:29:57.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:57 smithi029 ceph-osd[68010]: -- 172.21.15.29:0/1584395279 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56286c3d91e0 con 0x56286c377c00 2024-09-17T09:29:57.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:57 smithi029 bash[67990]: 2ff250ddb1d9df1dee5642100252b078f51d77b65c1b46cae55128b2ad38ae8f 2024-09-17T09:29:57.298 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:57 smithi029 systemd[1]: Started Ceph osd.1 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:29:57.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:57 smithi029 ceph-mon[58326]: Health check failed: Degraded data redundancy: 128/999 objects degraded (12.813%), 20 pgs degraded (PG_DEGRADED) 2024-09-17T09:29:57.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:57 smithi103 ceph-mon[49967]: Health check failed: Degraded data redundancy: 128/999 objects degraded (12.813%), 20 pgs degraded (PG_DEGRADED) 2024-09-17T09:29:57.798 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:29:57 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1[68006]: 2024-09-17T09:29:57.541+0000 7f330559f640 -1 Falling back to public interface 2024-09-17T09:29:58.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:58 smithi103 ceph-mon[49967]: pgmap v74: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2024-09-17T09:29:58.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:58 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:58.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:29:58 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:58.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:58 smithi029 ceph-mon[58326]: pgmap v74: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2024-09-17T09:29:58.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:58 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:58.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:29:58 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:58.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:58 smithi029 ceph-mon[60489]: pgmap v74: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2024-09-17T09:29:58.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:58 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:29:58.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:29:58 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pgmap v75: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2024-09-17T09:30:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 128/999 objects degraded (12.813%), 20 pgs degraded 2024-09-17T09:30:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:30:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:30:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:30:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: [WRN] OSD_DOWN: 1 osds down 2024-09-17T09:30:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: osd.1 (root=default,host=smithi029) is down 2024-09-17T09:30:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: [WRN] PG_DEGRADED: Degraded data redundancy: 128/999 objects degraded (12.813%), 20 pgs degraded 2024-09-17T09:30:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-09-17T09:30:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 2.9 is active+undersized+degraded, acting [2,3] 2024-09-17T09:30:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 2.a is active+undersized+degraded, acting [3,7] 2024-09-17T09:30:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 2.d is active+undersized+degraded, acting [4,3] 2024-09-17T09:30:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-09-17T09:30:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-09-17T09:30:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-09-17T09:30:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-09-17T09:30:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 3.a is active+undersized+degraded, acting [6,4] 2024-09-17T09:30:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 3.e is active+undersized+degraded, acting [7,4] 2024-09-17T09:30:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-09-17T09:30:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-09-17T09:30:00.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-09-17T09:30:00.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-09-17T09:30:00.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-09-17T09:30:00.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 4.d is active+undersized+degraded, acting [4,2] 2024-09-17T09:30:00.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 4.e is active+undersized+degraded, acting [4,6] 2024-09-17T09:30:00.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-09-17T09:30:00.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 4.1f is active+undersized+degraded, acting [6,5] 2024-09-17T09:30:00.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:00 smithi103 ceph-mon[49967]: pg 6.1a is active+undersized+degraded, acting [4,5] 2024-09-17T09:30:00.508 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1[68006]: 2024-09-17T09:30:00.265+0000 7f330559f640 -1 osd.1 0 read_superblock omap replica is missing. 2024-09-17T09:30:00.509 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1[68006]: 2024-09-17T09:30:00.307+0000 7f330559f640 -1 osd.1 83 log_to_monitors true 2024-09-17T09:30:00.509 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pgmap v75: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2024-09-17T09:30:00.509 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 128/999 objects degraded (12.813%), 20 pgs degraded 2024-09-17T09:30:00.509 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:30:00.509 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:30:00.509 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:30:00.509 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: [WRN] OSD_DOWN: 1 osds down 2024-09-17T09:30:00.509 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: osd.1 (root=default,host=smithi029) is down 2024-09-17T09:30:00.509 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: [WRN] PG_DEGRADED: Degraded data redundancy: 128/999 objects degraded (12.813%), 20 pgs degraded 2024-09-17T09:30:00.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-09-17T09:30:00.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 2.9 is active+undersized+degraded, acting [2,3] 2024-09-17T09:30:00.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 2.a is active+undersized+degraded, acting [3,7] 2024-09-17T09:30:00.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 2.d is active+undersized+degraded, acting [4,3] 2024-09-17T09:30:00.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-09-17T09:30:00.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-09-17T09:30:00.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-09-17T09:30:00.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-09-17T09:30:00.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 3.a is active+undersized+degraded, acting [6,4] 2024-09-17T09:30:00.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 3.e is active+undersized+degraded, acting [7,4] 2024-09-17T09:30:00.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-09-17T09:30:00.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-09-17T09:30:00.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-09-17T09:30:00.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-09-17T09:30:00.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-09-17T09:30:00.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 4.d is active+undersized+degraded, acting [4,2] 2024-09-17T09:30:00.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 4.e is active+undersized+degraded, acting [4,6] 2024-09-17T09:30:00.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-09-17T09:30:00.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 4.1f is active+undersized+degraded, acting [6,5] 2024-09-17T09:30:00.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[60489]: pg 6.1a is active+undersized+degraded, acting [4,5] 2024-09-17T09:30:00.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pgmap v75: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2024-09-17T09:30:00.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 128/999 objects degraded (12.813%), 20 pgs degraded 2024-09-17T09:30:00.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:30:00.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:30:00.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:30:00.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: [WRN] OSD_DOWN: 1 osds down 2024-09-17T09:30:00.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: osd.1 (root=default,host=smithi029) is down 2024-09-17T09:30:00.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: [WRN] PG_DEGRADED: Degraded data redundancy: 128/999 objects degraded (12.813%), 20 pgs degraded 2024-09-17T09:30:00.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-09-17T09:30:00.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 2.9 is active+undersized+degraded, acting [2,3] 2024-09-17T09:30:00.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 2.a is active+undersized+degraded, acting [3,7] 2024-09-17T09:30:00.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 2.d is active+undersized+degraded, acting [4,3] 2024-09-17T09:30:00.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-09-17T09:30:00.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-09-17T09:30:00.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-09-17T09:30:00.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-09-17T09:30:00.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 3.a is active+undersized+degraded, acting [6,4] 2024-09-17T09:30:00.513 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 3.e is active+undersized+degraded, acting [7,4] 2024-09-17T09:30:00.513 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-09-17T09:30:00.513 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-09-17T09:30:00.513 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-09-17T09:30:00.513 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-09-17T09:30:00.513 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-09-17T09:30:00.513 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 4.d is active+undersized+degraded, acting [4,2] 2024-09-17T09:30:00.513 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 4.e is active+undersized+degraded, acting [4,6] 2024-09-17T09:30:00.513 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-09-17T09:30:00.513 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 4.1f is active+undersized+degraded, acting [6,5] 2024-09-17T09:30:00.513 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:00 smithi029 ceph-mon[58326]: pg 6.1a is active+undersized+degraded, acting [4,5] 2024-09-17T09:30:01.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:01 smithi029 ceph-mon[58326]: from='osd.1 [v2:172.21.15.29:6810/1636111794,v1:172.21.15.29:6811/1636111794]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:30:01.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:01 smithi029 ceph-mon[58326]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:30:01.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:01 smithi029 ceph-mon[58326]: from='client.54403 172.21.15.29:0/1503125451' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:01.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:01 smithi029 ceph-mon[58326]: from='client.54409 172.21.15.29:0/809525443' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:30:01.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:01 smithi029 ceph-mon[60489]: from='osd.1 [v2:172.21.15.29:6810/1636111794,v1:172.21.15.29:6811/1636111794]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:30:01.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:01 smithi029 ceph-mon[60489]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:30:01.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:01 smithi029 ceph-mon[60489]: from='client.54403 172.21.15.29:0/1503125451' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:01.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:01 smithi029 ceph-mon[60489]: from='client.54409 172.21.15.29:0/809525443' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:30:01.299 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:30:01 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1[68006]: 2024-09-17T09:30:01.188+0000 7f32fd349640 -1 osd.1 83 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:30:01.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:01 smithi103 ceph-mon[49967]: from='osd.1 [v2:172.21.15.29:6810/1636111794,v1:172.21.15.29:6811/1636111794]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:30:01.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:01 smithi103 ceph-mon[49967]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T09:30:01.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:01 smithi103 ceph-mon[49967]: from='client.54403 172.21.15.29:0/1503125451' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:01.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:01 smithi103 ceph-mon[49967]: from='client.54409 172.21.15.29:0/809525443' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: access denied 2024-09-17T09:30:02.446 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:02 smithi029 ceph-mon[58326]: pgmap v76: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2024-09-17T09:30:02.446 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:02 smithi029 ceph-mon[58326]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T09:30:02.446 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:02 smithi029 ceph-mon[58326]: osdmap e86: 8 total, 7 up, 8 in 2024-09-17T09:30:02.446 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:02 smithi029 ceph-mon[58326]: from='osd.1 [v2:172.21.15.29:6810/1636111794,v1:172.21.15.29:6811/1636111794]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:02.446 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:02 smithi029 ceph-mon[58326]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:02.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:02 smithi029 ceph-mon[60489]: pgmap v76: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2024-09-17T09:30:02.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:02 smithi029 ceph-mon[60489]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T09:30:02.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:02 smithi029 ceph-mon[60489]: osdmap e86: 8 total, 7 up, 8 in 2024-09-17T09:30:02.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:02 smithi029 ceph-mon[60489]: from='osd.1 [v2:172.21.15.29:6810/1636111794,v1:172.21.15.29:6811/1636111794]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:02.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:02 smithi029 ceph-mon[60489]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:02 smithi103 ceph-mon[49967]: pgmap v76: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2024-09-17T09:30:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:02 smithi103 ceph-mon[49967]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T09:30:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:02 smithi103 ceph-mon[49967]: osdmap e86: 8 total, 7 up, 8 in 2024-09-17T09:30:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:02 smithi103 ceph-mon[49967]: from='osd.1 [v2:172.21.15.29:6810/1636111794,v1:172.21.15.29:6811/1636111794]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:02 smithi103 ceph-mon[49967]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:03.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:03 smithi103 ceph-mon[49967]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:30:03.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:03 smithi103 ceph-mon[49967]: osd.1 [v2:172.21.15.29:6810/1636111794,v1:172.21.15.29:6811/1636111794] boot 2024-09-17T09:30:03.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:03 smithi103 ceph-mon[49967]: osdmap e87: 8 total, 8 up, 8 in 2024-09-17T09:30:03.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:03 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:30:03.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:03 smithi103 ceph-mon[49967]: osdmap e88: 8 total, 8 up, 8 in 2024-09-17T09:30:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:03 smithi029 ceph-mon[60489]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:30:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:03 smithi029 ceph-mon[60489]: osd.1 [v2:172.21.15.29:6810/1636111794,v1:172.21.15.29:6811/1636111794] boot 2024-09-17T09:30:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:03 smithi029 ceph-mon[60489]: osdmap e87: 8 total, 8 up, 8 in 2024-09-17T09:30:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:03 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:30:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:03 smithi029 ceph-mon[60489]: osdmap e88: 8 total, 8 up, 8 in 2024-09-17T09:30:03.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:03 smithi029 ceph-mon[58326]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:30:03.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:03 smithi029 ceph-mon[58326]: osd.1 [v2:172.21.15.29:6810/1636111794,v1:172.21.15.29:6811/1636111794] boot 2024-09-17T09:30:03.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:03 smithi029 ceph-mon[58326]: osdmap e87: 8 total, 8 up, 8 in 2024-09-17T09:30:03.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:03 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T09:30:03.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:03 smithi029 ceph-mon[58326]: osdmap e88: 8 total, 8 up, 8 in 2024-09-17T09:30:04.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:04 smithi103 ceph-mon[49967]: pgmap v79: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2024-09-17T09:30:04.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:04 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:04.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:04 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:04.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:04 smithi029 ceph-mon[58326]: pgmap v79: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2024-09-17T09:30:04.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:04 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:04.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:04 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:04.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:04 smithi029 ceph-mon[60489]: pgmap v79: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 128/999 objects degraded (12.813%) 2024-09-17T09:30:04.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:04 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:04.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:04 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:05.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:05 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:05.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:05 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:05.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:05 smithi029 ceph-mon[58326]: pgmap v81: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 128/999 objects degraded (12.813%) 2024-09-17T09:30:05.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:05 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:05.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:05 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:05.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:05 smithi029 ceph-mon[60489]: pgmap v81: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 128/999 objects degraded (12.813%) 2024-09-17T09:30:05.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:05 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:05 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:05 smithi103 ceph-mon[49967]: pgmap v81: 161 pgs: 43 active+undersized, 20 active+undersized+degraded, 98 active+clean; 604 KiB data, 139 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 128/999 objects degraded (12.813%) 2024-09-17T09:30:07.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:07 smithi029 ceph-mon[58326]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 128/999 objects degraded (12.813%), 20 pgs degraded) 2024-09-17T09:30:07.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:07 smithi029 ceph-mon[60489]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 128/999 objects degraded (12.813%), 20 pgs degraded) 2024-09-17T09:30:07.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:07 smithi103 ceph-mon[49967]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 128/999 objects degraded (12.813%), 20 pgs degraded) 2024-09-17T09:30:08.193 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[58326]: pgmap v82: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:08.193 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:08.194 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:08.194 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:08.194 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:30:08.194 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:08.194 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:30:08.194 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:08.195 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:08.195 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:08.195 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T09:30:08.195 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[60489]: pgmap v82: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:08.196 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:08.196 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:08.196 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:08.196 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:30:08.196 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:08.196 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:30:08.197 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:08.197 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:08.197 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:08.197 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T09:30:08.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:08 smithi103 ceph-mon[49967]: pgmap v82: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:08.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:08.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:08.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:08.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:30:08.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:08.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:30:08.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:08.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:08.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:08.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T09:30:09.180 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:30:09.180 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[58326]: pgmap v83: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:09.180 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[58326]: pgmap v84: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:09.180 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T09:30:09.180 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[58326]: Upgrade: osd.2 is safe to restart 2024-09-17T09:30:09.180 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:09.180 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T09:30:09.180 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:09.180 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:30:09.180 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[60489]: pgmap v83: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:09.181 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[60489]: pgmap v84: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:09.181 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T09:30:09.181 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[60489]: Upgrade: osd.2 is safe to restart 2024-09-17T09:30:09.181 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:09.181 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T09:30:09.181 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:09 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:09.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:09 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:30:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:09 smithi103 ceph-mon[49967]: pgmap v83: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:09 smithi103 ceph-mon[49967]: pgmap v84: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:09 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T09:30:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:09 smithi103 ceph-mon[49967]: Upgrade: osd.2 is safe to restart 2024-09-17T09:30:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:09 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:09 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T09:30:09.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:09 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:10.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:10 smithi029 ceph-mon[58326]: Upgrade: Updating osd.2 2024-09-17T09:30:10.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:10 smithi029 ceph-mon[58326]: Deploying daemon osd.2 on smithi029 2024-09-17T09:30:10.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:10 smithi029 ceph-mon[60489]: Upgrade: Updating osd.2 2024-09-17T09:30:10.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:10 smithi029 ceph-mon[60489]: Deploying daemon osd.2 on smithi029 2024-09-17T09:30:10.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:10 smithi103 ceph-mon[49967]: Upgrade: Updating osd.2 2024-09-17T09:30:10.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:10 smithi103 ceph-mon[49967]: Deploying daemon osd.2 on smithi029 2024-09-17T09:30:11.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:11 smithi029 ceph-mon[58326]: pgmap v85: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:11.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:11 smithi029 ceph-mon[60489]: pgmap v85: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:11.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:11 smithi103 ceph-mon[49967]: pgmap v85: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:12.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:11 smithi029 systemd[1]: Stopping Ceph osd.2 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:30:12.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.2[34048]: debug 2024-09-17T09:30:11.959+0000 7f7964661700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T09:30:12.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.2[34048]: debug 2024-09-17T09:30:11.959+0000 7f7964661700 -1 osd.2 88 *** Got signal Terminated *** 2024-09-17T09:30:12.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:11 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.2[34048]: debug 2024-09-17T09:30:11.959+0000 7f7964661700 -1 osd.2 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:30:12.299 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:11 smithi029 podman[70109]: 2024-09-17 09:30:11.990990375 +0000 UTC m=+0.128953289 container died e9678d2f1685906fa7878ff283b556108a1832630fe5afbba23ddcd138556069 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.2, org.label-schema.name=CentOS Base Image, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-16.2.0) 2024-09-17T09:30:12.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:12 smithi029 podman[70109]: 2024-09-17 09:30:12.422094932 +0000 UTC m=+0.560057826 container cleanup e9678d2f1685906fa7878ff283b556108a1832630fe5afbba23ddcd138556069 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.2, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, RELEASE=HEAD, ceph=True, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=-16.2.0) 2024-09-17T09:30:12.642 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:12 smithi029 bash[70109]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.2 2024-09-17T09:30:12.908 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:12 smithi029 podman[70124]: 2024-09-17 09:30:12.64158755 +0000 UTC m=+0.645533240 container remove e9678d2f1685906fa7878ff283b556108a1832630fe5afbba23ddcd138556069 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, org.label-schema.build-date=20201204) 2024-09-17T09:30:13.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: pgmap v86: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:13.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 failed (root=default,host=smithi029) (connection refused reported by osd.7) 2024-09-17T09:30:13.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.6 2024-09-17T09:30:13.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.0 2024-09-17T09:30:13.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.0 2024-09-17T09:30:13.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.1 2024-09-17T09:30:13.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.1 2024-09-17T09:30:13.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.6 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.5 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.5 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.6 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.6 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.0 2024-09-17T09:30:13.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.1 2024-09-17T09:30:13.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.0 2024-09-17T09:30:13.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.1 2024-09-17T09:30:13.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.5 2024-09-17T09:30:13.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.5 2024-09-17T09:30:13.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.255 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[58326]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.255 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: pgmap v86: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:13.255 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 failed (root=default,host=smithi029) (connection refused reported by osd.7) 2024-09-17T09:30:13.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.6 2024-09-17T09:30:13.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.0 2024-09-17T09:30:13.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.0 2024-09-17T09:30:13.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.1 2024-09-17T09:30:13.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.1 2024-09-17T09:30:13.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.6 2024-09-17T09:30:13.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.5 2024-09-17T09:30:13.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.5 2024-09-17T09:30:13.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.6 2024-09-17T09:30:13.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.6 2024-09-17T09:30:13.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.0 2024-09-17T09:30:13.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.1 2024-09-17T09:30:13.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.0 2024-09-17T09:30:13.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.1 2024-09-17T09:30:13.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.5 2024-09-17T09:30:13.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.5 2024-09-17T09:30:13.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:13 smithi029 ceph-mon[60489]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: pgmap v86: 161 pgs: 161 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:13.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 failed (root=default,host=smithi029) (connection refused reported by osd.7) 2024-09-17T09:30:13.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.6 2024-09-17T09:30:13.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.0 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.0 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.1 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.1 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.6 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.5 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.5 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.6 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.6 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.0 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.1 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.0 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.1 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.5 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.5 2024-09-17T09:30:13.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.7 2024-09-17T09:30:13.509 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.4 2024-09-17T09:30:13.509 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:13 smithi103 ceph-mon[49967]: osd.2 reported immediately failed by osd.3 2024-09-17T09:30:14.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:14 smithi029 ceph-mon[58326]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:30:14.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:14 smithi029 ceph-mon[58326]: osdmap e89: 8 total, 7 up, 8 in 2024-09-17T09:30:14.277 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:14 smithi029 ceph-mon[60489]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:30:14.277 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:14 smithi029 ceph-mon[60489]: osdmap e89: 8 total, 7 up, 8 in 2024-09-17T09:30:14.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:14 smithi029 podman[70350]: 2024-09-17 09:30:13.908986168 +0000 UTC m=+0.022419494 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:30:14.277 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:14 smithi029 podman[70350]: 2024-09-17 09:30:14.027368778 +0000 UTC m=+0.140802088 container create 80714cf5f5bcfafcad863c8ada9032d082b53cbb66c59e4e0fc500c517465856 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-17T09:30:14.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:14 smithi103 ceph-mon[49967]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:30:14.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:14 smithi103 ceph-mon[49967]: osdmap e89: 8 total, 7 up, 8 in 2024-09-17T09:30:14.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:14 smithi029 podman[70350]: 2024-09-17 09:30:14.275814214 +0000 UTC m=+0.389247527 container init 80714cf5f5bcfafcad863c8ada9032d082b53cbb66c59e4e0fc500c517465856 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-17T09:30:14.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:14 smithi029 podman[70350]: 2024-09-17 09:30:14.281070397 +0000 UTC m=+0.394503705 container start 80714cf5f5bcfafcad863c8ada9032d082b53cbb66c59e4e0fc500c517465856 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-17T09:30:14.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:14 smithi029 podman[70350]: 2024-09-17 09:30:14.384838836 +0000 UTC m=+0.498272153 container attach 80714cf5f5bcfafcad863c8ada9032d082b53cbb66c59e4e0fc500c517465856 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-17T09:30:14.539 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:14 smithi029 podman[70350]: 2024-09-17 09:30:14.477204767 +0000 UTC m=+0.590638075 container died 80714cf5f5bcfafcad863c8ada9032d082b53cbb66c59e4e0fc500c517465856 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-17T09:30:14.539 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:30:15.208 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:30:15.208 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (3m) 11s ago 8m 19.1M - 0.20.0 0881eb8f169f 7bd158a6d65c 2024-09-17T09:30:15.208 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (3m) 95s ago 8m 30.5M - 6.7.4 557c83e11646 e361816f7c14 2024-09-17T09:30:15.208 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 unknown 11s ago 7m - - 2024-09-17T09:30:15.208 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (2m) 95s ago 12m 436M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:30:15.208 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (3m) 11s ago 13m 482M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:30:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (2m) 11s ago 13m 46.9M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:30:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (106s) 95s ago 12m 24.3M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 da050285f124 2024-09-17T09:30:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (2m) 11s ago 12m 43.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 0d3d1a65e2c9 2024-09-17T09:30:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (4m) 11s ago 9m 15.0M - 0.18.1 e5a616e4b9cf 34bfa263169f 2024-09-17T09:30:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (3m) 95s ago 8m 8866k - 0.18.1 e5a616e4b9cf 1adf1b5baad1 2024-09-17T09:30:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (53s) 11s ago 12m 51.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 4c456435000b 2024-09-17T09:30:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (18s) 11s ago 11m 12.1M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2ff250ddb1d9 2024-09-17T09:30:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (11m) 11s ago 11m 45.1M 4096M 16.2.0 24ecd6d5f14c e9678d2f1685 2024-09-17T09:30:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (11m) 11s ago 11m 51.2M 4096M 16.2.0 24ecd6d5f14c dcf7122a7d99 2024-09-17T09:30:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (10m) 95s ago 10m 49.1M 4096M 16.2.0 24ecd6d5f14c 048c13025d32 2024-09-17T09:30:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (10m) 95s ago 10m 48.0M 4096M 16.2.0 24ecd6d5f14c 10a64919fbc9 2024-09-17T09:30:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (10m) 95s ago 10m 46.7M 4096M 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:30:15.209 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (9m) 95s ago 9m 52.8M 4096M 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:30:15.210 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 unknown 95s ago 9m - - 2024-09-17T09:30:15.210 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (7m) 11s ago 7m 41.3M - 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:30:15.210 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (7m) 95s ago 7m 40.7M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:30:15.210 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (7m) 11s ago 7m 39.1M - 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:30:15.210 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (7m) 95s ago 7m 38.0M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:30:15.339 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:15 smithi029 ceph-mon[60489]: pgmap v88: 161 pgs: 12 stale+active+clean, 149 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:15.339 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:15 smithi029 ceph-mon[60489]: osdmap e90: 8 total, 7 up, 8 in 2024-09-17T09:30:15.339 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:15 smithi029 ceph-mon[60489]: from='client.54427 172.21.15.29:0/3844686179' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:15.339 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:15 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:15.339 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:15 smithi029 ceph-mon[60489]: from='client.44556 172.21.15.29:0/3118847175' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:30:15.340 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:15 smithi029 ceph-mon[58326]: pgmap v88: 161 pgs: 12 stale+active+clean, 149 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:15.340 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:15 smithi029 ceph-mon[58326]: osdmap e90: 8 total, 7 up, 8 in 2024-09-17T09:30:15.340 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:15 smithi029 ceph-mon[58326]: from='client.54427 172.21.15.29:0/3844686179' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:15.340 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:15 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:15.340 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:15 smithi029 ceph-mon[58326]: from='client.44556 172.21.15.29:0/3118847175' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:30:15.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:15 smithi103 ceph-mon[49967]: pgmap v88: 161 pgs: 12 stale+active+clean, 149 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:15.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:15 smithi103 ceph-mon[49967]: osdmap e90: 8 total, 7 up, 8 in 2024-09-17T09:30:15.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:15 smithi103 ceph-mon[49967]: from='client.54427 172.21.15.29:0/3844686179' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:15.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:15 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:15.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:15 smithi103 ceph-mon[49967]: from='client.44556 172.21.15.29:0/3118847175' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:30:15.616 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:30:15.616 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:30:15.616 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:30:15.617 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:30:15.617 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:30:15.617 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:30:15.617 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:30:15.617 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:30:15.617 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 5, 2024-09-17T09:30:15.617 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:30:15.617 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:30:15.618 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:30:15.618 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:30:15.618 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:30:15.618 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:30:15.618 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 9, 2024-09-17T09:30:15.618 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 7 2024-09-17T09:30:15.618 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:30:15.618 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:30:15.958 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:30:15.958 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:30:15.958 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:30:15.958 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:30:15.959 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [ 2024-09-17T09:30:15.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mon", 2024-09-17T09:30:15.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr" 2024-09-17T09:30:15.959 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:30:15.959 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "7/23 daemons upgraded", 2024-09-17T09:30:15.959 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T09:30:15.959 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:30:15.959 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:30:16.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:16 smithi029 ceph-mon[58326]: from='client.54433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:16.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:16 smithi029 ceph-mon[58326]: from='client.54445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:16.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:16 smithi029 ceph-mon[58326]: from='client.44568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:16.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:16 smithi029 ceph-mon[58326]: from='client.54457 172.21.15.29:0/87686191' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:16.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:16 smithi029 ceph-mon[60489]: from='client.54433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:16.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:16 smithi029 ceph-mon[60489]: from='client.54445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:16.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:16 smithi029 ceph-mon[60489]: from='client.44568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:16.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:16 smithi029 ceph-mon[60489]: from='client.54457 172.21.15.29:0/87686191' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:16.363 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 2 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 50/999 objects degraded (5.005%), 9 pgs degraded 2024-09-17T09:30:16.363 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:30:16.363 INFO:teuthology.orchestra.run.smithi029.stdout: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:30:16.363 INFO:teuthology.orchestra.run.smithi029.stdout: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:30:16.364 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-17T09:30:16.364 INFO:teuthology.orchestra.run.smithi029.stdout: osd.2 (root=default,host=smithi029) is down 2024-09-17T09:30:16.364 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 50/999 objects degraded (5.005%), 9 pgs degraded 2024-09-17T09:30:16.364 INFO:teuthology.orchestra.run.smithi029.stdout: pg 2.e is active+undersized+degraded, acting [3,7] 2024-09-17T09:30:16.364 INFO:teuthology.orchestra.run.smithi029.stdout: pg 2.17 is active+undersized+degraded, acting [6,5] 2024-09-17T09:30:16.364 INFO:teuthology.orchestra.run.smithi029.stdout: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-09-17T09:30:16.364 INFO:teuthology.orchestra.run.smithi029.stdout: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-09-17T09:30:16.364 INFO:teuthology.orchestra.run.smithi029.stdout: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-09-17T09:30:16.364 INFO:teuthology.orchestra.run.smithi029.stdout: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-09-17T09:30:16.364 INFO:teuthology.orchestra.run.smithi029.stdout: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-09-17T09:30:16.364 INFO:teuthology.orchestra.run.smithi029.stdout: pg 4.9 is active+undersized+degraded, acting [4,3] 2024-09-17T09:30:16.365 INFO:teuthology.orchestra.run.smithi029.stdout: pg 4.d is active+undersized+degraded, acting [4,1] 2024-09-17T09:30:16.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:16 smithi103 ceph-mon[49967]: from='client.54433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:16.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:16 smithi103 ceph-mon[49967]: from='client.54445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:16.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:16 smithi103 ceph-mon[49967]: from='client.44568 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:16.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:16 smithi103 ceph-mon[49967]: from='client.54457 172.21.15.29:0/87686191' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:16.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:16 smithi029 podman[70440]: 2024-09-17 09:30:16.326528657 +0000 UTC m=+1.843264505 container remove 80714cf5f5bcfafcad863c8ada9032d082b53cbb66c59e4e0fc500c517465856 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-17T09:30:16.775 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:16 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.2.service: Deactivated successfully. 2024-09-17T09:30:16.776 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:16 smithi029 systemd[1]: Stopped Ceph osd.2 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:30:16.776 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:16 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.2.service: Consumed 1.175s CPU time. 2024-09-17T09:30:16.776 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:16 smithi029 systemd[1]: Starting Ceph osd.2 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:30:17.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:16 smithi029 podman[70707]: 2024-09-17 09:30:16.774628468 +0000 UTC m=+0.105671914 container create b49d826ed9d2dc704ea7589823174aad18950be148f8d0c6ca44dd3d2b502068 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-17T09:30:17.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:16 smithi029 podman[70707]: 2024-09-17 09:30:16.685411817 +0000 UTC m=+0.016455263 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:30:17.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:16 smithi029 podman[70707]: 2024-09-17 09:30:16.987301291 +0000 UTC m=+0.318344737 container init b49d826ed9d2dc704ea7589823174aad18950be148f8d0c6ca44dd3d2b502068 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:30:17.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:16 smithi029 podman[70707]: 2024-09-17 09:30:16.990857543 +0000 UTC m=+0.321900992 container start b49d826ed9d2dc704ea7589823174aad18950be148f8d0c6ca44dd3d2b502068 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:30:17.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-mon[60489]: pgmap v90: 161 pgs: 23 active+undersized, 4 stale+active+clean, 9 active+undersized+degraded, 125 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 50/999 objects degraded (5.005%) 2024-09-17T09:30:17.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-mon[60489]: from='client.54463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:17.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-mon[60489]: Health check failed: Degraded data redundancy: 50/999 objects degraded (5.005%), 9 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:17.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-mon[60489]: from='client.54469 172.21.15.29:0/3702386439' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:30:17.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-mon[58326]: pgmap v90: 161 pgs: 23 active+undersized, 4 stale+active+clean, 9 active+undersized+degraded, 125 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 50/999 objects degraded (5.005%) 2024-09-17T09:30:17.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-mon[58326]: from='client.54463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:17.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-mon[58326]: Health check failed: Degraded data redundancy: 50/999 objects degraded (5.005%), 9 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:17.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-mon[58326]: from='client.54469 172.21.15.29:0/3702386439' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:30:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 podman[70707]: 2024-09-17 09:30:17.080965843 +0000 UTC m=+0.412009298 container attach b49d826ed9d2dc704ea7589823174aad18950be148f8d0c6ca44dd3d2b502068 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-activate, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:30:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-activate[70722]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T09:30:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 bash[70707]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T09:30:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-activate[70722]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T09:30:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 bash[70707]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T09:30:17.320 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-activate[70722]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T09:30:17.321 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 bash[70707]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T09:30:17.406 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:17 smithi103 ceph-mon[49967]: pgmap v90: 161 pgs: 23 active+undersized, 4 stale+active+clean, 9 active+undersized+degraded, 125 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 50/999 objects degraded (5.005%) 2024-09-17T09:30:17.406 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:17 smithi103 ceph-mon[49967]: from='client.54463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:17.406 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:17 smithi103 ceph-mon[49967]: Health check failed: Degraded data redundancy: 50/999 objects degraded (5.005%), 9 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:17.406 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:17 smithi103 ceph-mon[49967]: from='client.54469 172.21.15.29:0/3702386439' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:30:17.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-activate[70722]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T09:30:17.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 bash[70707]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T09:30:17.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-activate[70722]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-17T09:30:17.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 bash[70707]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-17T09:30:17.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-activate[70722]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T09:30:17.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 bash[70707]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T09:30:17.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-activate[70722]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-17T09:30:17.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 bash[70707]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-17T09:30:17.646 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:17 smithi029 podman[70707]: 2024-09-17 09:30:17.34567724 +0000 UTC m=+0.676720686 container died b49d826ed9d2dc704ea7589823174aad18950be148f8d0c6ca44dd3d2b502068 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True) 2024-09-17T09:30:18.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:18 smithi029 podman[70836]: 2024-09-17 09:30:18.015703562 +0000 UTC m=+0.659042793 container remove b49d826ed9d2dc704ea7589823174aad18950be148f8d0c6ca44dd3d2b502068 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-activate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-17T09:30:18.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:18 smithi029 podman[70894]: 2024-09-17 09:30:18.136616815 +0000 UTC m=+0.016775532 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:30:18.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:18 smithi029 podman[70894]: 2024-09-17 09:30:18.250719533 +0000 UTC m=+0.130878250 container create 7899f3c3c2f4823a0c652fd0ce241da3101e6d219cfa1e66b4463ee8d7eb8a4f (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:30:18.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:18 smithi029 podman[70894]: 2024-09-17 09:30:18.497255291 +0000 UTC m=+0.377414015 container init 7899f3c3c2f4823a0c652fd0ce241da3101e6d219cfa1e66b4463ee8d7eb8a4f (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-17T09:30:18.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:18 smithi029 podman[70894]: 2024-09-17 09:30:18.500668328 +0000 UTC m=+0.380827044 container start 7899f3c3c2f4823a0c652fd0ce241da3101e6d219cfa1e66b4463ee8d7eb8a4f (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-17T09:30:18.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:18 smithi029 ceph-osd[70912]: -- 172.21.15.29:0/742068558 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5585bd9ab1e0 con 0x5585be72a000 2024-09-17T09:30:18.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:18 smithi029 bash[70894]: 7899f3c3c2f4823a0c652fd0ce241da3101e6d219cfa1e66b4463ee8d7eb8a4f 2024-09-17T09:30:18.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:18 smithi029 systemd[1]: Started Ceph osd.2 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:30:19.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:19 smithi029 ceph-mon[60489]: pgmap v91: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T09:30:19.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:19 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:19.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:19 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:19.298 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:19 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2[70908]: 2024-09-17T09:30:19.044+0000 7fd6cc65d640 -1 Falling back to public interface 2024-09-17T09:30:19.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:19 smithi029 ceph-mon[58326]: pgmap v91: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T09:30:19.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:19 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:19.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:19 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:19.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:19 smithi103 ceph-mon[49967]: pgmap v91: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T09:30:19.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:19 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:19.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:19 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:21.273 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:21 smithi029 ceph-mon[60489]: pgmap v92: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T09:30:21.273 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:21 smithi029 ceph-mon[58326]: pgmap v92: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T09:30:21.431 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:21 smithi103 ceph-mon[49967]: pgmap v92: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T09:30:21.534 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2[70908]: 2024-09-17T09:30:21.512+0000 7fd6cc65d640 -1 osd.2 0 read_superblock omap replica is missing. 2024-09-17T09:30:21.534 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:21 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2[70908]: 2024-09-17T09:30:21.533+0000 7fd6cc65d640 -1 osd.2 88 log_to_monitors true 2024-09-17T09:30:22.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:22 smithi029 ceph-mon[58326]: from='osd.2 [v2:172.21.15.29:6818/3154444148,v1:172.21.15.29:6819/3154444148]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:30:22.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:22 smithi029 ceph-mon[58326]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:30:22.323 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:22 smithi029 ceph-mon[60489]: from='osd.2 [v2:172.21.15.29:6818/3154444148,v1:172.21.15.29:6819/3154444148]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:30:22.323 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:22 smithi029 ceph-mon[60489]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:30:22.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:22 smithi103 ceph-mon[49967]: from='osd.2 [v2:172.21.15.29:6818/3154444148,v1:172.21.15.29:6819/3154444148]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:30:22.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:22 smithi103 ceph-mon[49967]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T09:30:22.797 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:30:22 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2[70908]: 2024-09-17T09:30:22.342+0000 7fd6c03ff640 -1 osd.2 88 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:30:23.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:23 smithi103 ceph-mon[49967]: pgmap v93: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T09:30:23.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:23 smithi103 ceph-mon[49967]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T09:30:23.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:23 smithi103 ceph-mon[49967]: osdmap e91: 8 total, 7 up, 8 in 2024-09-17T09:30:23.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:23 smithi103 ceph-mon[49967]: from='osd.2 [v2:172.21.15.29:6818/3154444148,v1:172.21.15.29:6819/3154444148]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:23.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:23 smithi103 ceph-mon[49967]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:23.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:23 smithi103 ceph-mon[49967]: from='osd.2 ' entity='osd.2' 2024-09-17T09:30:23.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:23 smithi103 ceph-mon[49967]: Health check update: Degraded data redundancy: 88/999 objects degraded (8.809%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:23.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[60489]: pgmap v93: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T09:30:23.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[60489]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T09:30:23.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[60489]: osdmap e91: 8 total, 7 up, 8 in 2024-09-17T09:30:23.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[60489]: from='osd.2 [v2:172.21.15.29:6818/3154444148,v1:172.21.15.29:6819/3154444148]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:23.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[60489]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:23.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[60489]: from='osd.2 ' entity='osd.2' 2024-09-17T09:30:23.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[60489]: Health check update: Degraded data redundancy: 88/999 objects degraded (8.809%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:23.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[58326]: pgmap v93: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 604 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 88/999 objects degraded (8.809%) 2024-09-17T09:30:23.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[58326]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T09:30:23.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[58326]: osdmap e91: 8 total, 7 up, 8 in 2024-09-17T09:30:23.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[58326]: from='osd.2 [v2:172.21.15.29:6818/3154444148,v1:172.21.15.29:6819/3154444148]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:23.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[58326]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:23.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[58326]: from='osd.2 ' entity='osd.2' 2024-09-17T09:30:23.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:23 smithi029 ceph-mon[58326]: Health check update: Degraded data redundancy: 88/999 objects degraded (8.809%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:24.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:24.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:24.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[58326]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:30:24.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[58326]: osd.2 [v2:172.21.15.29:6818/3154444148,v1:172.21.15.29:6819/3154444148] boot 2024-09-17T09:30:24.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[58326]: osdmap e92: 8 total, 8 up, 8 in 2024-09-17T09:30:24.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:30:24.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:24.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:24.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:24.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:24.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[60489]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:30:24.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[60489]: osd.2 [v2:172.21.15.29:6818/3154444148,v1:172.21.15.29:6819/3154444148] boot 2024-09-17T09:30:24.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[60489]: osdmap e92: 8 total, 8 up, 8 in 2024-09-17T09:30:24.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:30:24.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:24.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:24 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:24.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:24 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:24 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:24 smithi103 ceph-mon[49967]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:30:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:24 smithi103 ceph-mon[49967]: osd.2 [v2:172.21.15.29:6818/3154444148,v1:172.21.15.29:6819/3154444148] boot 2024-09-17T09:30:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:24 smithi103 ceph-mon[49967]: osdmap e92: 8 total, 8 up, 8 in 2024-09-17T09:30:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:24 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T09:30:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:24 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:24 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:25.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:25 smithi029 ceph-mon[58326]: pgmap v96: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 604 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 88/999 objects degraded (8.809%) 2024-09-17T09:30:25.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:25 smithi029 ceph-mon[58326]: osdmap e93: 8 total, 8 up, 8 in 2024-09-17T09:30:25.176 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:25 smithi029 ceph-mon[60489]: pgmap v96: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 604 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 88/999 objects degraded (8.809%) 2024-09-17T09:30:25.176 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:25 smithi029 ceph-mon[60489]: osdmap e93: 8 total, 8 up, 8 in 2024-09-17T09:30:25.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:25 smithi103 ceph-mon[49967]: pgmap v96: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 604 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 88/999 objects degraded (8.809%) 2024-09-17T09:30:25.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:25 smithi103 ceph-mon[49967]: osdmap e93: 8 total, 8 up, 8 in 2024-09-17T09:30:27.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:27 smithi029 ceph-mon[58326]: pgmap v98: 161 pgs: 18 active+undersized, 5 active+undersized+degraded, 138 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 40/999 objects degraded (4.004%) 2024-09-17T09:30:27.243 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:27 smithi029 ceph-mon[60489]: pgmap v98: 161 pgs: 18 active+undersized, 5 active+undersized+degraded, 138 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 40/999 objects degraded (4.004%) 2024-09-17T09:30:27.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:27 smithi103 ceph-mon[49967]: pgmap v98: 161 pgs: 18 active+undersized, 5 active+undersized+degraded, 138 active+clean; 604 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 40/999 objects degraded (4.004%) 2024-09-17T09:30:29.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: pgmap v99: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:29.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:29.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:29.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:29.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:30:29.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:30:29.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: pgmap v100: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:29.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: pgmap v101: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:29.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:29.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: Health check update: Degraded data redundancy: 40/999 objects degraded (4.004%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:29.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:30:29.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:29.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:29.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:29.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T09:30:29.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T09:30:29.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: Upgrade: osd.3 is safe to restart 2024-09-17T09:30:29.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/999 objects degraded (4.004%), 5 pgs degraded) 2024-09-17T09:30:29.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:28 smithi103 ceph-mon[49967]: from='client.54493 172.21.15.29:0/730848537' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:29.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: pgmap v99: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:29.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:29.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:30:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:30:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: pgmap v100: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: pgmap v101: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: Health check update: Degraded data redundancy: 40/999 objects degraded (4.004%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:30:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T09:30:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T09:30:29.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: Upgrade: osd.3 is safe to restart 2024-09-17T09:30:29.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/999 objects degraded (4.004%), 5 pgs degraded) 2024-09-17T09:30:29.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[58326]: from='client.54493 172.21.15.29:0/730848537' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: pgmap v99: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:30:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:30:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: pgmap v100: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:29.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: pgmap v101: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:29.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:29.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: Health check update: Degraded data redundancy: 40/999 objects degraded (4.004%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:29.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:30:29.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:29.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:29.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:29.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T09:30:29.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T09:30:29.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: Upgrade: osd.3 is safe to restart 2024-09-17T09:30:29.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/999 objects degraded (4.004%), 5 pgs degraded) 2024-09-17T09:30:29.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:28 smithi029 ceph-mon[60489]: from='client.54493 172.21.15.29:0/730848537' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:29.771 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:29 smithi029 ceph-mon[58326]: from='client.44616 172.21.15.29:0/87601929' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:30:29.771 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:29 smithi029 ceph-mon[58326]: Upgrade: Updating osd.3 2024-09-17T09:30:29.771 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:29 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:29.771 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:29 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T09:30:29.771 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:29 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:29.771 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:29 smithi029 ceph-mon[58326]: Deploying daemon osd.3 on smithi029 2024-09-17T09:30:29.771 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:29 smithi029 ceph-mon[60489]: from='client.44616 172.21.15.29:0/87601929' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:30:29.772 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:29 smithi029 ceph-mon[60489]: Upgrade: Updating osd.3 2024-09-17T09:30:29.772 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:29 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:29.772 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:29 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T09:30:29.772 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:29 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:29.772 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:29 smithi029 ceph-mon[60489]: Deploying daemon osd.3 on smithi029 2024-09-17T09:30:30.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:29 smithi103 ceph-mon[49967]: from='client.44616 172.21.15.29:0/87601929' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:30:30.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:29 smithi103 ceph-mon[49967]: Upgrade: Updating osd.3 2024-09-17T09:30:30.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:29 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:30.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:29 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T09:30:30.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:29 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:30.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:29 smithi103 ceph-mon[49967]: Deploying daemon osd.3 on smithi029 2024-09-17T09:30:30.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:30 smithi029 ceph-mon[60489]: pgmap v102: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:30.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:30 smithi029 ceph-mon[58326]: pgmap v102: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:31.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:30 smithi103 ceph-mon[49967]: pgmap v102: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:32.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:32 smithi029 ceph-mon[58326]: pgmap v103: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 139 B/s rd, 0 op/s 2024-09-17T09:30:32.888 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:32 smithi029 ceph-mon[60489]: pgmap v103: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 139 B/s rd, 0 op/s 2024-09-17T09:30:33.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:32 smithi103 ceph-mon[49967]: pgmap v103: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 139 B/s rd, 0 op/s 2024-09-17T09:30:33.548 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:33 smithi029 systemd[1]: Stopping Ceph osd.3 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:30:33.548 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.3[36462]: debug 2024-09-17T09:30:33.411+0000 7f944112b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T09:30:33.548 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.3[36462]: debug 2024-09-17T09:30:33.411+0000 7f944112b700 -1 osd.3 93 *** Got signal Terminated *** 2024-09-17T09:30:33.548 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.3[36462]: debug 2024-09-17T09:30:33.411+0000 7f944112b700 -1 osd.3 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:30:33.548 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:33 smithi029 podman[73014]: 2024-09-17 09:30:33.452841784 +0000 UTC m=+0.154026536 container died dcf7122a7d99aca4d4b893be1e3cc29912501642181dd770cb303bd02084fce9 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.3, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=-16.2.0, GIT_CLEAN=True) 2024-09-17T09:30:34.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.7 2024-09-17T09:30:34.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 failed (root=default,host=smithi029) (connection refused reported by osd.7) 2024-09-17T09:30:34.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.6 2024-09-17T09:30:34.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.4 2024-09-17T09:30:34.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.7 2024-09-17T09:30:34.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.5 2024-09-17T09:30:34.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.1 2024-09-17T09:30:34.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.6 2024-09-17T09:30:34.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.4 2024-09-17T09:30:34.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.0 2024-09-17T09:30:34.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.1 2024-09-17T09:30:34.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.2 2024-09-17T09:30:34.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.5 2024-09-17T09:30:34.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.0 2024-09-17T09:30:34.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:33 smithi103 ceph-mon[49967]: osd.3 reported immediately failed by osd.2 2024-09-17T09:30:34.048 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:33 smithi029 podman[73014]: 2024-09-17 09:30:33.847145421 +0000 UTC m=+0.548330174 container cleanup dcf7122a7d99aca4d4b893be1e3cc29912501642181dd770cb303bd02084fce9 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.3, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=-16.2.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) 2024-09-17T09:30:34.048 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:33 smithi029 bash[73014]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.3 2024-09-17T09:30:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.7 2024-09-17T09:30:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 failed (root=default,host=smithi029) (connection refused reported by osd.7) 2024-09-17T09:30:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.6 2024-09-17T09:30:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.4 2024-09-17T09:30:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.7 2024-09-17T09:30:34.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.5 2024-09-17T09:30:34.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.1 2024-09-17T09:30:34.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.6 2024-09-17T09:30:34.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.4 2024-09-17T09:30:34.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.0 2024-09-17T09:30:34.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.1 2024-09-17T09:30:34.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.2 2024-09-17T09:30:34.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.5 2024-09-17T09:30:34.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.0 2024-09-17T09:30:34.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[58326]: osd.3 reported immediately failed by osd.2 2024-09-17T09:30:34.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.7 2024-09-17T09:30:34.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 failed (root=default,host=smithi029) (connection refused reported by osd.7) 2024-09-17T09:30:34.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.6 2024-09-17T09:30:34.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.4 2024-09-17T09:30:34.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.7 2024-09-17T09:30:34.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.5 2024-09-17T09:30:34.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.1 2024-09-17T09:30:34.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.6 2024-09-17T09:30:34.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.4 2024-09-17T09:30:34.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.0 2024-09-17T09:30:34.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.1 2024-09-17T09:30:34.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.2 2024-09-17T09:30:34.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.5 2024-09-17T09:30:34.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.0 2024-09-17T09:30:34.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:33 smithi029 ceph-mon[60489]: osd.3 reported immediately failed by osd.2 2024-09-17T09:30:34.548 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:34 smithi029 podman[73028]: 2024-09-17 09:30:34.084060429 +0000 UTC m=+0.622801857 container remove dcf7122a7d99aca4d4b893be1e3cc29912501642181dd770cb303bd02084fce9 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.3, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=HEAD, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image) 2024-09-17T09:30:34.933 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:34 smithi029 ceph-mon[60489]: pgmap v104: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 124 B/s rd, 0 op/s 2024-09-17T09:30:34.934 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:34 smithi029 ceph-mon[60489]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:30:34.934 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:34 smithi029 ceph-mon[60489]: osdmap e94: 8 total, 7 up, 8 in 2024-09-17T09:30:34.934 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:34 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:34.934 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:34 smithi029 ceph-mon[58326]: pgmap v104: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 124 B/s rd, 0 op/s 2024-09-17T09:30:34.934 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:34 smithi029 ceph-mon[58326]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:30:34.934 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:34 smithi029 ceph-mon[58326]: osdmap e94: 8 total, 7 up, 8 in 2024-09-17T09:30:34.934 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:34 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:34.935 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:34 smithi029 podman[73133]: 2024-09-17 09:30:34.489213596 +0000 UTC m=+0.016648003 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:30:34.935 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:34 smithi029 podman[73133]: 2024-09-17 09:30:34.594420385 +0000 UTC m=+0.121854783 container create 03e38447d0292bdb8b381d694ee7b2b1f77be302cb4fb7760ff5a4c9b30251e9 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-deactivate, ceph=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-17T09:30:34.935 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:34 smithi029 podman[73133]: 2024-09-17 09:30:34.824375787 +0000 UTC m=+0.351810197 container init 03e38447d0292bdb8b381d694ee7b2b1f77be302cb4fb7760ff5a4c9b30251e9 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-17T09:30:34.935 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:34 smithi029 podman[73133]: 2024-09-17 09:30:34.82780544 +0000 UTC m=+0.355239837 container start 03e38447d0292bdb8b381d694ee7b2b1f77be302cb4fb7760ff5a4c9b30251e9 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True) 2024-09-17T09:30:34.935 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:34 smithi029 podman[73133]: 2024-09-17 09:30:34.933165723 +0000 UTC m=+0.460600132 container attach 03e38447d0292bdb8b381d694ee7b2b1f77be302cb4fb7760ff5a4c9b30251e9 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=) 2024-09-17T09:30:35.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:34 smithi103 ceph-mon[49967]: pgmap v104: 161 pgs: 161 active+clean; 604 KiB data, 166 MiB used, 715 GiB / 715 GiB avail; 124 B/s rd, 0 op/s 2024-09-17T09:30:35.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:34 smithi103 ceph-mon[49967]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:30:35.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:34 smithi103 ceph-mon[49967]: osdmap e94: 8 total, 7 up, 8 in 2024-09-17T09:30:35.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:34 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:35.298 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:35 smithi029 podman[73133]: 2024-09-17 09:30:35.022535187 +0000 UTC m=+0.549969582 container died 03e38447d0292bdb8b381d694ee7b2b1f77be302cb4fb7760ff5a4c9b30251e9 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T09:30:36.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:35 smithi103 ceph-mon[49967]: osdmap e95: 8 total, 7 up, 8 in 2024-09-17T09:30:36.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:35 smithi029 ceph-mon[58326]: osdmap e95: 8 total, 7 up, 8 in 2024-09-17T09:30:36.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:35 smithi029 ceph-mon[60489]: osdmap e95: 8 total, 7 up, 8 in 2024-09-17T09:30:36.049 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:35 smithi029 podman[73157]: 2024-09-17 09:30:35.634206472 +0000 UTC m=+0.605495796 container remove 03e38447d0292bdb8b381d694ee7b2b1f77be302cb4fb7760ff5a4c9b30251e9 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:30:36.049 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:35 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.3.service: Deactivated successfully. 2024-09-17T09:30:36.049 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:35 smithi029 systemd[1]: Stopped Ceph osd.3 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:30:36.049 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:35 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.3.service: Consumed 1.128s CPU time. 2024-09-17T09:30:36.049 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:35 smithi029 systemd[1]: Starting Ceph osd.3 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:30:36.385 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 podman[73257]: 2024-09-17 09:30:35.993327859 +0000 UTC m=+0.016754626 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:30:36.385 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 podman[73257]: 2024-09-17 09:30:36.130329741 +0000 UTC m=+0.153756534 container create 45f47dfbd232c1c4e7e305ae24172c33cf988a2a23aa67bf189f2e0026a6d665 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:30:36.695 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 podman[73257]: 2024-09-17 09:30:36.384889859 +0000 UTC m=+0.408316618 container init 45f47dfbd232c1c4e7e305ae24172c33cf988a2a23aa67bf189f2e0026a6d665 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-activate, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-17T09:30:36.696 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 podman[73257]: 2024-09-17 09:30:36.388439499 +0000 UTC m=+0.411866256 container start 45f47dfbd232c1c4e7e305ae24172c33cf988a2a23aa67bf189f2e0026a6d665 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:30:36.696 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 podman[73257]: 2024-09-17 09:30:36.485297426 +0000 UTC m=+0.508724221 container attach 45f47dfbd232c1c4e7e305ae24172c33cf988a2a23aa67bf189f2e0026a6d665 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-activate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:30:37.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:36 smithi103 ceph-mon[49967]: pgmap v107: 161 pgs: 22 active+undersized, 12 stale+active+clean, 11 active+undersized+degraded, 116 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 59/999 objects degraded (5.906%) 2024-09-17T09:30:37.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:36 smithi103 ceph-mon[49967]: Health check failed: Degraded data redundancy: 59/999 objects degraded (5.906%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:37.034 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:36 smithi029 ceph-mon[58326]: pgmap v107: 161 pgs: 22 active+undersized, 12 stale+active+clean, 11 active+undersized+degraded, 116 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 59/999 objects degraded (5.906%) 2024-09-17T09:30:37.034 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:36 smithi029 ceph-mon[58326]: Health check failed: Degraded data redundancy: 59/999 objects degraded (5.906%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:37.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:36 smithi029 ceph-mon[60489]: pgmap v107: 161 pgs: 22 active+undersized, 12 stale+active+clean, 11 active+undersized+degraded, 116 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 59/999 objects degraded (5.906%) 2024-09-17T09:30:37.035 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:36 smithi029 ceph-mon[60489]: Health check failed: Degraded data redundancy: 59/999 objects degraded (5.906%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:37.035 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-activate[73272]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T09:30:37.035 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 bash[73257]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T09:30:37.035 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-activate[73272]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T09:30:37.035 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 bash[73257]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T09:30:37.035 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-activate[73272]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T09:30:37.035 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 bash[73257]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T09:30:37.035 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-activate[73272]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T09:30:37.035 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 bash[73257]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T09:30:37.036 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-activate[73272]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-17T09:30:37.036 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 bash[73257]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-17T09:30:37.036 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-activate[73272]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T09:30:37.036 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 bash[73257]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T09:30:37.036 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-activate[73272]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-17T09:30:37.036 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 bash[73257]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-17T09:30:37.036 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:36 smithi029 podman[73257]: 2024-09-17 09:30:36.744127489 +0000 UTC m=+0.767554254 container died 45f47dfbd232c1c4e7e305ae24172c33cf988a2a23aa67bf189f2e0026a6d665 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-activate, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-17T09:30:37.688 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:37 smithi029 podman[73392]: 2024-09-17 09:30:37.352636658 +0000 UTC m=+0.602818314 container remove 45f47dfbd232c1c4e7e305ae24172c33cf988a2a23aa67bf189f2e0026a6d665 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-activate, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:30:37.688 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:37 smithi029 podman[73453]: 2024-09-17 09:30:37.47386928 +0000 UTC m=+0.016514051 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:30:37.688 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:37 smithi029 podman[73453]: 2024-09-17 09:30:37.580968973 +0000 UTC m=+0.123613727 container create 94b84aca13ce200a1801bf455dbf6a3454f469efcaad660f30530d743f466865 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-17T09:30:38.048 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:37 smithi029 podman[73453]: 2024-09-17 09:30:37.785368139 +0000 UTC m=+0.328012905 container init 94b84aca13ce200a1801bf455dbf6a3454f469efcaad660f30530d743f466865 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-17T09:30:38.048 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:37 smithi029 podman[73453]: 2024-09-17 09:30:37.788699129 +0000 UTC m=+0.331343883 container start 94b84aca13ce200a1801bf455dbf6a3454f469efcaad660f30530d743f466865 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T09:30:38.048 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:37 smithi029 ceph-osd[73473]: -- 172.21.15.29:0/2154122528 <== mon.1 v2:172.21.15.29:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56362406d1e0 con 0x56362400bc00 2024-09-17T09:30:38.048 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:37 smithi029 bash[73453]: 94b84aca13ce200a1801bf455dbf6a3454f469efcaad660f30530d743f466865 2024-09-17T09:30:38.048 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:37 smithi029 systemd[1]: Started Ceph osd.3 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:30:38.798 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3[73469]: 2024-09-17T09:30:38.326+0000 7f06a7379640 -1 Falling back to public interface 2024-09-17T09:30:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:38 smithi103 ceph-mon[49967]: pgmap v108: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 135/999 objects degraded (13.514%) 2024-09-17T09:30:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:39.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:38 smithi029 ceph-mon[58326]: pgmap v108: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 135/999 objects degraded (13.514%) 2024-09-17T09:30:39.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:39.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:38 smithi029 ceph-mon[60489]: pgmap v108: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 135/999 objects degraded (13.514%) 2024-09-17T09:30:39.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:39.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:41.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:40 smithi029 ceph-mon[60489]: pgmap v109: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 135/999 objects degraded (13.514%) 2024-09-17T09:30:41.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:40 smithi029 ceph-mon[58326]: pgmap v109: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 135/999 objects degraded (13.514%) 2024-09-17T09:30:41.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:40 smithi103 ceph-mon[49967]: pgmap v109: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 135/999 objects degraded (13.514%) 2024-09-17T09:30:41.368 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3[73469]: 2024-09-17T09:30:41.061+0000 7f06a7379640 -1 osd.3 0 read_superblock omap replica is missing. 2024-09-17T09:30:41.368 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:41 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3[73469]: 2024-09-17T09:30:41.087+0000 7f06a7379640 -1 osd.3 93 log_to_monitors true 2024-09-17T09:30:42.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:41 smithi103 ceph-mon[49967]: from='osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T09:30:42.259 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:41 smithi029 ceph-mon[60489]: from='osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T09:30:42.259 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:41 smithi029 ceph-mon[58326]: from='osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T09:30:42.548 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3[73469]: 2024-09-17T09:30:42.258+0000 7f069b11b640 -1 osd.3 93 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:30:43.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[58326]: pgmap v110: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 135/999 objects degraded (13.514%) 2024-09-17T09:30:43.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[58326]: from='osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T09:30:43.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[58326]: osdmap e96: 8 total, 7 up, 8 in 2024-09-17T09:30:43.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[58326]: from='osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:43.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[58326]: from='osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492]' entity='osd.3' 2024-09-17T09:30:43.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[58326]: from='client.44640 172.21.15.29:0/3861392185' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:43.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[58326]: Health check update: Degraded data redundancy: 135/999 objects degraded (13.514%), 23 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:43.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[60489]: pgmap v110: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 135/999 objects degraded (13.514%) 2024-09-17T09:30:43.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[60489]: from='osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T09:30:43.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[60489]: osdmap e96: 8 total, 7 up, 8 in 2024-09-17T09:30:43.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[60489]: from='osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:43.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[60489]: from='osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492]' entity='osd.3' 2024-09-17T09:30:43.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[60489]: from='client.44640 172.21.15.29:0/3861392185' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:43.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:42 smithi029 ceph-mon[60489]: Health check update: Degraded data redundancy: 135/999 objects degraded (13.514%), 23 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:43.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:42 smithi103 ceph-mon[49967]: pgmap v110: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 604 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 135/999 objects degraded (13.514%) 2024-09-17T09:30:43.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:42 smithi103 ceph-mon[49967]: from='osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T09:30:43.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:42 smithi103 ceph-mon[49967]: osdmap e96: 8 total, 7 up, 8 in 2024-09-17T09:30:43.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:42 smithi103 ceph-mon[49967]: from='osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-09-17T09:30:43.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:42 smithi103 ceph-mon[49967]: from='osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492]' entity='osd.3' 2024-09-17T09:30:43.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:42 smithi103 ceph-mon[49967]: from='client.44640 172.21.15.29:0/3861392185' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:43.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:42 smithi103 ceph-mon[49967]: Health check update: Degraded data redundancy: 135/999 objects degraded (13.514%), 23 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:44.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[58326]: from='client.44646 172.21.15.29:0/98065125' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:30:44.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[58326]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:30:44.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[58326]: osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492] boot 2024-09-17T09:30:44.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[58326]: osdmap e97: 8 total, 8 up, 8 in 2024-09-17T09:30:44.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:30:44.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:44.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:44.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[60489]: from='client.44646 172.21.15.29:0/98065125' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:30:44.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[60489]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:30:44.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[60489]: osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492] boot 2024-09-17T09:30:44.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[60489]: osdmap e97: 8 total, 8 up, 8 in 2024-09-17T09:30:44.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:30:44.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:44.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:43 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:44.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:43 smithi103 ceph-mon[49967]: from='client.44646 172.21.15.29:0/98065125' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:30:44.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:43 smithi103 ceph-mon[49967]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:30:44.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:43 smithi103 ceph-mon[49967]: osd.3 [v2:172.21.15.29:6826/656719492,v1:172.21.15.29:6827/656719492] boot 2024-09-17T09:30:44.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:43 smithi103 ceph-mon[49967]: osdmap e97: 8 total, 8 up, 8 in 2024-09-17T09:30:44.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:43 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T09:30:44.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:43 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:44.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:43 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:45.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:44 smithi103 ceph-mon[49967]: pgmap v113: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 604 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 135/999 objects degraded (13.514%) 2024-09-17T09:30:45.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:44 smithi103 ceph-mon[49967]: osdmap e98: 8 total, 8 up, 8 in 2024-09-17T09:30:45.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:45.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:45.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:44 smithi029 ceph-mon[58326]: pgmap v113: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 604 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 135/999 objects degraded (13.514%) 2024-09-17T09:30:45.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:44 smithi029 ceph-mon[58326]: osdmap e98: 8 total, 8 up, 8 in 2024-09-17T09:30:45.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:45.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:45.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:44 smithi029 ceph-mon[60489]: pgmap v113: 161 pgs: 43 active+undersized, 23 active+undersized+degraded, 95 active+clean; 604 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 135/999 objects degraded (13.514%) 2024-09-17T09:30:45.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:44 smithi029 ceph-mon[60489]: osdmap e98: 8 total, 8 up, 8 in 2024-09-17T09:30:45.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:45.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:46.740 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:30:47.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:46 smithi103 ceph-mon[49967]: pgmap v115: 161 pgs: 31 active+undersized, 17 active+undersized+degraded, 113 active+clean; 604 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 107/999 objects degraded (10.711%) 2024-09-17T09:30:47.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:46 smithi029 ceph-mon[58326]: pgmap v115: 161 pgs: 31 active+undersized, 17 active+undersized+degraded, 113 active+clean; 604 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 107/999 objects degraded (10.711%) 2024-09-17T09:30:47.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:46 smithi029 ceph-mon[60489]: pgmap v115: 161 pgs: 31 active+undersized, 17 active+undersized+degraded, 113 active+clean; 604 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 107/999 objects degraded (10.711%) 2024-09-17T09:30:47.429 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:30:47.429 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (4m) 4s ago 9m 19.1M - 0.20.0 0881eb8f169f 7bd158a6d65c 2024-09-17T09:30:47.429 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (3m) 2m ago 8m 30.5M - 6.7.4 557c83e11646 e361816f7c14 2024-09-17T09:30:47.429 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 running (6s) 4s ago 8m - - 24ecd6d5f14c 8d25644a0887 2024-09-17T09:30:47.429 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (3m) 2m ago 12m 436M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:30:47.429 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (3m) 4s ago 14m 485M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:30:47.429 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (2m) 4s ago 14m 49.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:30:47.429 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (2m) 2m ago 13m 24.3M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 da050285f124 2024-09-17T09:30:47.429 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (2m) 4s ago 13m 45.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 0d3d1a65e2c9 2024-09-17T09:30:47.429 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (4m) 4s ago 9m 15.0M - 0.18.1 e5a616e4b9cf 34bfa263169f 2024-09-17T09:30:47.429 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (4m) 2m ago 9m 8866k - 0.18.1 e5a616e4b9cf 1adf1b5baad1 2024-09-17T09:30:47.430 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (85s) 4s ago 12m 52.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 4c456435000b 2024-09-17T09:30:47.430 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (50s) 4s ago 12m 44.3M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2ff250ddb1d9 2024-09-17T09:30:47.430 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (29s) 4s ago 11m 49.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7899f3c3c2f4 2024-09-17T09:30:47.430 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (9s) 4s ago 11m 12.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 94b84aca13ce 2024-09-17T09:30:47.430 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (11m) 2m ago 11m 49.1M 4096M 16.2.0 24ecd6d5f14c 048c13025d32 2024-09-17T09:30:47.430 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (11m) 2m ago 11m 48.0M 4096M 16.2.0 24ecd6d5f14c 10a64919fbc9 2024-09-17T09:30:47.430 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (10m) 2m ago 10m 46.7M 4096M 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:30:47.430 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (10m) 2m ago 10m 52.8M 4096M 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:30:47.430 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 unknown 2m ago 9m - - 2024-09-17T09:30:47.430 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (8m) 4s ago 8m 41.6M - 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:30:47.430 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (8m) 2m ago 8m 40.7M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:30:47.430 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (8m) 4s ago 8m 39.3M - 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:30:47.430 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (8m) 2m ago 8m 38.0M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:30:47.843 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:30:47.843 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:30:47.843 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:30:47.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:30:47.844 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:30:47.844 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:30:47.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:30:47.844 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:30:47.844 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4, 2024-09-17T09:30:47.844 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T09:30:47.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:30:47.844 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:30:47.844 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:30:47.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:30:47.844 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:30:47.844 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8, 2024-09-17T09:30:47.844 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 9 2024-09-17T09:30:47.845 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:30:47.845 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:30:48.190 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:30:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:30:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:30:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:30:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [ 2024-09-17T09:30:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "mon", 2024-09-17T09:30:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr" 2024-09-17T09:30:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:30:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "9/23 daemons upgraded", 2024-09-17T09:30:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T09:30:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:30:48.191 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:30:48.191 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:47 smithi029 ceph-mon[60489]: from='client.54535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:48.191 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:47 smithi029 ceph-mon[60489]: from='client.54541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:48.191 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:47 smithi029 ceph-mon[60489]: Health check update: Degraded data redundancy: 107/999 objects degraded (10.711%), 17 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:48.191 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:47 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/2170913535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:48.192 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:47 smithi029 ceph-mon[58326]: from='client.54535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:48.192 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:47 smithi029 ceph-mon[58326]: from='client.54541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:48.192 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:47 smithi029 ceph-mon[58326]: Health check update: Degraded data redundancy: 107/999 objects degraded (10.711%), 17 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:48.192 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:47 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/2170913535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:48.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:47 smithi103 ceph-mon[49967]: from='client.54535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:48.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:47 smithi103 ceph-mon[49967]: from='client.54541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:48.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:47 smithi103 ceph-mon[49967]: Health check update: Degraded data redundancy: 107/999 objects degraded (10.711%), 17 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:48.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:47 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/2170913535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:48.610 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 2 failed cephadm daemon(s) 2024-09-17T09:30:48.610 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-09-17T09:30:48.610 INFO:teuthology.orchestra.run.smithi029.stdout: daemon iscsi.foo.smithi029.gellkh on smithi029 is in unknown state 2024-09-17T09:30:48.611 INFO:teuthology.orchestra.run.smithi029.stdout: daemon prometheus.a on smithi103 is in unknown state 2024-09-17T09:30:49.080 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: from='client.54547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:49.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: pgmap v116: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:49.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 107/999 objects degraded (10.711%), 17 pgs degraded) 2024-09-17T09:30:49.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: from='client.54559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:49.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:49.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:49.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:49.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:30:49.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:49.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:30:49.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:49.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:49.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:49.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T09:30:49.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:48 smithi103 ceph-mon[49967]: from='client.44682 172.21.15.29:0/1905461221' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:30:49.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: from='client.54547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:49.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: pgmap v116: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:49.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 107/999 objects degraded (10.711%), 17 pgs degraded) 2024-09-17T09:30:49.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: from='client.54559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:49.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:49.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:49.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:49.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:30:49.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:49.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:30:49.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:49.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:49.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:49.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T09:30:49.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[58326]: from='client.44682 172.21.15.29:0/1905461221' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:30:49.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: from='client.54547 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:49.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: pgmap v116: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:49.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 107/999 objects degraded (10.711%), 17 pgs degraded) 2024-09-17T09:30:49.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: from='client.54559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:30:49.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:49.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:49.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:49.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:30:49.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:49.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:30:49.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:49.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:49.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:30:49.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T09:30:49.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:48 smithi029 ceph-mon[60489]: from='client.44682 172.21.15.29:0/1905461221' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:30:50.217 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:49 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:30:50.217 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:49 smithi103 ceph-mon[49967]: pgmap v117: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:50.217 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:49 smithi103 ceph-mon[49967]: pgmap v118: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:50.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:49 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T09:30:50.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:49 smithi103 ceph-mon[49967]: Upgrade: osd.4 is safe to restart 2024-09-17T09:30:50.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:49 smithi103 ceph-mon[49967]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:30:50.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:49 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:50.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:49 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T09:30:50.218 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:49 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:30:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[58326]: pgmap v117: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[58326]: pgmap v118: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T09:30:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[58326]: Upgrade: osd.4 is safe to restart 2024-09-17T09:30:50.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[58326]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:30:50.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:50.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T09:30:50.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:50.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:30:50.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[60489]: pgmap v117: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:50.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[60489]: pgmap v118: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:50.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T09:30:50.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[60489]: Upgrade: osd.4 is safe to restart 2024-09-17T09:30:50.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[60489]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:30:50.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:50.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T09:30:50.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:49 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:30:51.208 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:50 smithi103 ceph-mon[49967]: Upgrade: Updating osd.4 2024-09-17T09:30:51.208 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:50 smithi103 ceph-mon[49967]: Deploying daemon osd.4 on smithi103 2024-09-17T09:30:51.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:50 smithi029 ceph-mon[58326]: Upgrade: Updating osd.4 2024-09-17T09:30:51.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:50 smithi029 ceph-mon[58326]: Deploying daemon osd.4 on smithi103 2024-09-17T09:30:51.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:50 smithi029 ceph-mon[60489]: Upgrade: Updating osd.4 2024-09-17T09:30:51.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:50 smithi029 ceph-mon[60489]: Deploying daemon osd.4 on smithi103 2024-09-17T09:30:52.099 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:51 smithi103 ceph-mon[49967]: pgmap v119: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:52.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:51 smithi029 ceph-mon[58326]: pgmap v119: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:52.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:51 smithi029 ceph-mon[60489]: pgmap v119: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:53.012 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:52 smithi103 systemd[1]: Stopping Ceph osd.4 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:30:53.012 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.4[26604]: debug 2024-09-17T09:30:52.968+0000 7f0891c4f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T09:30:53.012 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.4[26604]: debug 2024-09-17T09:30:52.968+0000 7f0891c4f700 -1 osd.4 98 *** Got signal Terminated *** 2024-09-17T09:30:53.012 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.4[26604]: debug 2024-09-17T09:30:52.968+0000 7f0891c4f700 -1 osd.4 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:30:53.297 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:53 smithi103 podman[53069]: 2024-09-17 09:30:53.010842996 +0000 UTC m=+0.207723919 container died 048c13025d328d14932c52cb923f96e7d2129de25fb901d113ce9258462b786d (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.4, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=-16.2.0, ceph=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Dimitri Savineau , RELEASE=HEAD) 2024-09-17T09:30:53.638 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:53 smithi103 podman[53069]: 2024-09-17 09:30:53.403327989 +0000 UTC m=+0.600208905 container cleanup 048c13025d328d14932c52cb923f96e7d2129de25fb901d113ce9258462b786d (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.4, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, maintainer=Dimitri Savineau , RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-09-17T09:30:53.638 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:53 smithi103 bash[53069]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.4 2024-09-17T09:30:53.967 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:53 smithi103 podman[53081]: 2024-09-17 09:30:53.637928305 +0000 UTC m=+0.630763402 container remove 048c13025d328d14932c52cb923f96e7d2129de25fb901d113ce9258462b786d (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.4, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557) 2024-09-17T09:30:54.218 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:54 smithi103 podman[53195]: 2024-09-17 09:30:54.001337503 +0000 UTC m=+0.022134970 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:30:54.218 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:54 smithi103 podman[53195]: 2024-09-17 09:30:54.105660726 +0000 UTC m=+0.126458182 container create e78e089ae5c7614a7d2a5b78e534c076feaf96371e2739a2c32f324d4d025f11 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-17T09:30:54.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: pgmap v120: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:54.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.7 2024-09-17T09:30:54.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 failed (root=default,host=smithi103) (connection refused reported by osd.7) 2024-09-17T09:30:54.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.7 2024-09-17T09:30:54.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.3 2024-09-17T09:30:54.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.1 2024-09-17T09:30:54.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.6 2024-09-17T09:30:54.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.3 2024-09-17T09:30:54.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.5 2024-09-17T09:30:54.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.1 2024-09-17T09:30:54.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.6 2024-09-17T09:30:54.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.5 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.0 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.0 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.2 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.2 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.7 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.6 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.7 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.3 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.0 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.1 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.6 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.3 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.0 2024-09-17T09:30:54.220 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.1 2024-09-17T09:30:54.221 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.2 2024-09-17T09:30:54.221 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.5 2024-09-17T09:30:54.221 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.2 2024-09-17T09:30:54.221 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:53 smithi103 ceph-mon[49967]: osd.4 reported immediately failed by osd.5 2024-09-17T09:30:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: pgmap v120: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.7 2024-09-17T09:30:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 failed (root=default,host=smithi103) (connection refused reported by osd.7) 2024-09-17T09:30:54.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.7 2024-09-17T09:30:54.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.3 2024-09-17T09:30:54.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.1 2024-09-17T09:30:54.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.6 2024-09-17T09:30:54.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.3 2024-09-17T09:30:54.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.5 2024-09-17T09:30:54.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.1 2024-09-17T09:30:54.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.6 2024-09-17T09:30:54.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.5 2024-09-17T09:30:54.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.0 2024-09-17T09:30:54.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.0 2024-09-17T09:30:54.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.2 2024-09-17T09:30:54.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.2 2024-09-17T09:30:54.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.7 2024-09-17T09:30:54.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.6 2024-09-17T09:30:54.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.7 2024-09-17T09:30:54.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.3 2024-09-17T09:30:54.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.0 2024-09-17T09:30:54.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.1 2024-09-17T09:30:54.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.6 2024-09-17T09:30:54.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.3 2024-09-17T09:30:54.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.0 2024-09-17T09:30:54.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.1 2024-09-17T09:30:54.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.2 2024-09-17T09:30:54.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.5 2024-09-17T09:30:54.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.2 2024-09-17T09:30:54.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[58326]: osd.4 reported immediately failed by osd.5 2024-09-17T09:30:54.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: pgmap v120: 161 pgs: 161 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:54.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.7 2024-09-17T09:30:54.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 failed (root=default,host=smithi103) (connection refused reported by osd.7) 2024-09-17T09:30:54.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.7 2024-09-17T09:30:54.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.3 2024-09-17T09:30:54.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.1 2024-09-17T09:30:54.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.6 2024-09-17T09:30:54.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.3 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.5 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.1 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.6 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.5 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.0 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.0 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.2 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.2 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.7 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.6 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.7 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.3 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.0 2024-09-17T09:30:54.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.1 2024-09-17T09:30:54.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.6 2024-09-17T09:30:54.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.3 2024-09-17T09:30:54.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.0 2024-09-17T09:30:54.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.1 2024-09-17T09:30:54.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.2 2024-09-17T09:30:54.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.5 2024-09-17T09:30:54.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.2 2024-09-17T09:30:54.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:53 smithi029 ceph-mon[60489]: osd.4 reported immediately failed by osd.5 2024-09-17T09:30:54.756 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:54 smithi103 podman[53195]: 2024-09-17 09:30:54.324969783 +0000 UTC m=+0.345767247 container init e78e089ae5c7614a7d2a5b78e534c076feaf96371e2739a2c32f324d4d025f11 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-17T09:30:54.756 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:54 smithi103 podman[53195]: 2024-09-17 09:30:54.328510663 +0000 UTC m=+0.349308114 container start e78e089ae5c7614a7d2a5b78e534c076feaf96371e2739a2c32f324d4d025f11 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-deactivate, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-17T09:30:54.756 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:54 smithi103 podman[53195]: 2024-09-17 09:30:54.408108268 +0000 UTC m=+0.428905748 container attach e78e089ae5c7614a7d2a5b78e534c076feaf96371e2739a2c32f324d4d025f11 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-17T09:30:54.756 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:54 smithi103 podman[53195]: 2024-09-17 09:30:54.515093689 +0000 UTC m=+0.535891140 container died e78e089ae5c7614a7d2a5b78e534c076feaf96371e2739a2c32f324d4d025f11 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T09:30:55.096 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:54 smithi103 ceph-mon[49967]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:30:55.096 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:54 smithi103 ceph-mon[49967]: osdmap e99: 8 total, 7 up, 8 in 2024-09-17T09:30:55.096 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:54 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:55.270 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:54 smithi029 ceph-mon[58326]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:30:55.270 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:54 smithi029 ceph-mon[58326]: osdmap e99: 8 total, 7 up, 8 in 2024-09-17T09:30:55.270 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:54 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:55.270 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:54 smithi029 ceph-mon[60489]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:30:55.271 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:54 smithi029 ceph-mon[60489]: osdmap e99: 8 total, 7 up, 8 in 2024-09-17T09:30:55.271 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:54 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:55.506 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:55 smithi103 podman[53217]: 2024-09-17 09:30:55.095619333 +0000 UTC m=+0.575014118 container remove e78e089ae5c7614a7d2a5b78e534c076feaf96371e2739a2c32f324d4d025f11 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-17T09:30:55.506 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:55 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.4.service: Deactivated successfully. 2024-09-17T09:30:55.506 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:55 smithi103 systemd[1]: Stopped Ceph osd.4 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:30:55.506 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:55 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.4.service: Consumed 1.130s CPU time. 2024-09-17T09:30:55.506 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:55 smithi103 systemd[1]: Starting Ceph osd.4 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:30:55.858 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:55 smithi103 podman[53316]: 2024-09-17 09:30:55.441916371 +0000 UTC m=+0.016266607 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:30:55.858 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:55 smithi103 podman[53316]: 2024-09-17 09:30:55.563584443 +0000 UTC m=+0.137934721 container create 24bcf6ccd891fd5ce7bf5978e8d967d741f94f0342922badb61aedbaba8fddf0 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-17T09:30:55.859 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:55 smithi103 podman[53316]: 2024-09-17 09:30:55.774307801 +0000 UTC m=+0.348658023 container init 24bcf6ccd891fd5ce7bf5978e8d967d741f94f0342922badb61aedbaba8fddf0 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-activate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:30:55.859 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:55 smithi103 podman[53316]: 2024-09-17 09:30:55.777861053 +0000 UTC m=+0.352211276 container start 24bcf6ccd891fd5ce7bf5978e8d967d741f94f0342922badb61aedbaba8fddf0 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-17T09:30:55.859 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:55 smithi103 podman[53316]: 2024-09-17 09:30:55.857713477 +0000 UTC m=+0.432063719 container attach 24bcf6ccd891fd5ce7bf5978e8d967d741f94f0342922badb61aedbaba8fddf0 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-17T09:30:56.122 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:55 smithi103 ceph-mon[49967]: pgmap v122: 161 pgs: 23 stale+active+clean, 138 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:56.122 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:55 smithi103 ceph-mon[49967]: osdmap e100: 8 total, 7 up, 8 in 2024-09-17T09:30:56.122 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-activate[53331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T09:30:56.122 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 bash[53316]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T09:30:56.122 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-activate[53331]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T09:30:56.123 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 bash[53316]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T09:30:56.123 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-activate[53331]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T09:30:56.123 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 bash[53316]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T09:30:56.123 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-activate[53331]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T09:30:56.123 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 bash[53316]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T09:30:56.123 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-activate[53331]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-17T09:30:56.123 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 bash[53316]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-17T09:30:56.123 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-activate[53331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T09:30:56.123 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 bash[53316]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T09:30:56.123 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-activate[53331]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-17T09:30:56.123 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 bash[53316]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-17T09:30:56.123 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 podman[53316]: 2024-09-17 09:30:56.12153782 +0000 UTC m=+0.695888057 container died 24bcf6ccd891fd5ce7bf5978e8d967d741f94f0342922badb61aedbaba8fddf0 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-activate, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-17T09:30:56.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:55 smithi029 ceph-mon[58326]: pgmap v122: 161 pgs: 23 stale+active+clean, 138 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:56.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:55 smithi029 ceph-mon[58326]: osdmap e100: 8 total, 7 up, 8 in 2024-09-17T09:30:56.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:55 smithi029 ceph-mon[60489]: pgmap v122: 161 pgs: 23 stale+active+clean, 138 active+clean; 604 KiB data, 199 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:30:56.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:55 smithi029 ceph-mon[60489]: osdmap e100: 8 total, 7 up, 8 in 2024-09-17T09:30:56.984 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:56 smithi103 podman[53448]: 2024-09-17 09:30:56.811868674 +0000 UTC m=+0.683803421 container remove 24bcf6ccd891fd5ce7bf5978e8d967d741f94f0342922badb61aedbaba8fddf0 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-17T09:30:57.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:56 smithi029 ceph-mon[58326]: from='client.54583 172.21.15.29:0/3199691092' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:57.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:56 smithi029 ceph-mon[58326]: from='client.54589 172.21.15.29:0/2807892416' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:30:57.220 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:56 smithi029 ceph-mon[60489]: from='client.54583 172.21.15.29:0/3199691092' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:57.220 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:56 smithi029 ceph-mon[60489]: from='client.54589 172.21.15.29:0/2807892416' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:30:57.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:56 smithi103 ceph-mon[49967]: from='client.54583 172.21.15.29:0/3199691092' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:30:57.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:56 smithi103 ceph-mon[49967]: from='client.54589 172.21.15.29:0/2807892416' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:30:57.256 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:57 smithi103 podman[53508]: 2024-09-17 09:30:56.919455635 +0000 UTC m=+0.015574408 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:30:57.256 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:57 smithi103 podman[53508]: 2024-09-17 09:30:57.054610227 +0000 UTC m=+0.150728983 container create 2306402255eaed03c119220983081f76e5aa258b2e21b327f59e0ef1090c06ea (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-17T09:30:57.756 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:57 smithi103 podman[53508]: 2024-09-17 09:30:57.279606861 +0000 UTC m=+0.375725636 container init 2306402255eaed03c119220983081f76e5aa258b2e21b327f59e0ef1090c06ea (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-17T09:30:57.756 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:57 smithi103 podman[53508]: 2024-09-17 09:30:57.282790336 +0000 UTC m=+0.378909094 container start 2306402255eaed03c119220983081f76e5aa258b2e21b327f59e0ef1090c06ea (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD) 2024-09-17T09:30:57.756 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:57 smithi103 ceph-osd[53528]: -- 172.21.15.103:0/676800816 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5638e4f2d1e0 con 0x5638e4ecbc00 2024-09-17T09:30:57.756 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:57 smithi103 bash[53508]: 2306402255eaed03c119220983081f76e5aa258b2e21b327f59e0ef1090c06ea 2024-09-17T09:30:57.756 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:57 smithi103 systemd[1]: Started Ceph osd.4 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:30:58.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:57 smithi029 ceph-mon[58326]: pgmap v124: 161 pgs: 26 active+undersized, 12 stale+active+clean, 16 active+undersized+degraded, 107 active+clean; 604 KiB data, 203 MiB used, 715 GiB / 715 GiB avail; 109/999 objects degraded (10.911%) 2024-09-17T09:30:58.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:57 smithi029 ceph-mon[58326]: Health check failed: Degraded data redundancy: 109/999 objects degraded (10.911%), 16 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:58.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:57 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:58.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:57 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:58.106 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:57 smithi029 ceph-mon[60489]: pgmap v124: 161 pgs: 26 active+undersized, 12 stale+active+clean, 16 active+undersized+degraded, 107 active+clean; 604 KiB data, 203 MiB used, 715 GiB / 715 GiB avail; 109/999 objects degraded (10.911%) 2024-09-17T09:30:58.106 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:57 smithi029 ceph-mon[60489]: Health check failed: Degraded data redundancy: 109/999 objects degraded (10.911%), 16 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:58.106 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:57 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:58.106 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:57 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:58.112 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:30:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:30:57.830+0000 7f95b1de8640 -1 Falling back to public interface 2024-09-17T09:30:58.112 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:57 smithi103 ceph-mon[49967]: pgmap v124: 161 pgs: 26 active+undersized, 12 stale+active+clean, 16 active+undersized+degraded, 107 active+clean; 604 KiB data, 203 MiB used, 715 GiB / 715 GiB avail; 109/999 objects degraded (10.911%) 2024-09-17T09:30:58.112 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:57 smithi103 ceph-mon[49967]: Health check failed: Degraded data redundancy: 109/999 objects degraded (10.911%), 16 pgs degraded (PG_DEGRADED) 2024-09-17T09:30:58.112 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:57 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:30:58.112 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:57 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:00.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:30:59 smithi103 ceph-mon[49967]: pgmap v125: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 604 KiB data, 203 MiB used, 715 GiB / 715 GiB avail; 167/999 objects degraded (16.717%) 2024-09-17T09:31:00.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:30:59 smithi029 ceph-mon[58326]: pgmap v125: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 604 KiB data, 203 MiB used, 715 GiB / 715 GiB avail; 167/999 objects degraded (16.717%) 2024-09-17T09:31:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:30:59 smithi029 ceph-mon[60489]: pgmap v125: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 604 KiB data, 203 MiB used, 715 GiB / 715 GiB avail; 167/999 objects degraded (16.717%) 2024-09-17T09:31:00.601 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:31:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:31:00.569+0000 7f95b1de8640 -1 osd.4 0 read_superblock omap replica is missing. 2024-09-17T09:31:00.601 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:31:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:31:00.599+0000 7f95b1de8640 -1 osd.4 98 log_to_monitors true 2024-09-17T09:31:01.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:00 smithi103 ceph-mon[49967]: from='osd.4 [v2:172.21.15.103:6800/2627911661,v1:172.21.15.103:6801/2627911661]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:31:01.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:00 smithi103 ceph-mon[49967]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:31:01.256 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:31:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:31:01.027+0000 7f95a9b92640 -1 osd.4 98 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:31:01.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:00 smithi029 ceph-mon[60489]: from='osd.4 [v2:172.21.15.103:6800/2627911661,v1:172.21.15.103:6801/2627911661]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:31:01.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:00 smithi029 ceph-mon[60489]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:31:01.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:00 smithi029 ceph-mon[58326]: from='osd.4 [v2:172.21.15.103:6800/2627911661,v1:172.21.15.103:6801/2627911661]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:31:01.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:00 smithi029 ceph-mon[58326]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T09:31:02.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[60489]: pgmap v126: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 604 KiB data, 203 MiB used, 715 GiB / 715 GiB avail; 167/999 objects degraded (16.717%) 2024-09-17T09:31:02.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[60489]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T09:31:02.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[60489]: osdmap e101: 8 total, 7 up, 8 in 2024-09-17T09:31:02.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[60489]: from='osd.4 [v2:172.21.15.103:6800/2627911661,v1:172.21.15.103:6801/2627911661]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:31:02.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[60489]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:31:02.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:02.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:02.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[58326]: pgmap v126: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 604 KiB data, 203 MiB used, 715 GiB / 715 GiB avail; 167/999 objects degraded (16.717%) 2024-09-17T09:31:02.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[58326]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T09:31:02.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[58326]: osdmap e101: 8 total, 7 up, 8 in 2024-09-17T09:31:02.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[58326]: from='osd.4 [v2:172.21.15.103:6800/2627911661,v1:172.21.15.103:6801/2627911661]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:31:02.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[58326]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:31:02.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:02.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:02 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:02.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:02 smithi103 ceph-mon[49967]: pgmap v126: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 604 KiB data, 203 MiB used, 715 GiB / 715 GiB avail; 167/999 objects degraded (16.717%) 2024-09-17T09:31:02.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:02 smithi103 ceph-mon[49967]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T09:31:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:02 smithi103 ceph-mon[49967]: osdmap e101: 8 total, 7 up, 8 in 2024-09-17T09:31:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:02 smithi103 ceph-mon[49967]: from='osd.4 [v2:172.21.15.103:6800/2627911661,v1:172.21.15.103:6801/2627911661]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:31:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:02 smithi103 ceph-mon[49967]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:31:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:02 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:02 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:03.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:03 smithi103 ceph-mon[49967]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:31:03.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:03 smithi103 ceph-mon[49967]: osd.4 [v2:172.21.15.103:6800/2627911661,v1:172.21.15.103:6801/2627911661] boot 2024-09-17T09:31:03.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:03 smithi103 ceph-mon[49967]: osdmap e102: 8 total, 8 up, 8 in 2024-09-17T09:31:03.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:03 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:31:03.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:03 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:03.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:03 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:03.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:03 smithi103 ceph-mon[49967]: Health check update: Degraded data redundancy: 167/999 objects degraded (16.717%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T09:31:03.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:03 smithi103 ceph-mon[49967]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T09:31:03.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[60489]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:31:03.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[60489]: osd.4 [v2:172.21.15.103:6800/2627911661,v1:172.21.15.103:6801/2627911661] boot 2024-09-17T09:31:03.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[60489]: osdmap e102: 8 total, 8 up, 8 in 2024-09-17T09:31:03.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:31:03.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:03.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:03.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[60489]: Health check update: Degraded data redundancy: 167/999 objects degraded (16.717%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T09:31:03.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[60489]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T09:31:03.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[58326]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:31:03.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[58326]: osd.4 [v2:172.21.15.103:6800/2627911661,v1:172.21.15.103:6801/2627911661] boot 2024-09-17T09:31:03.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[58326]: osdmap e102: 8 total, 8 up, 8 in 2024-09-17T09:31:03.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T09:31:03.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:03.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:03.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[58326]: Health check update: Degraded data redundancy: 167/999 objects degraded (16.717%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T09:31:03.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:03 smithi029 ceph-mon[58326]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T09:31:04.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:04 smithi103 ceph-mon[49967]: pgmap v129: 161 pgs: 3 peering, 39 active+undersized, 26 active+undersized+degraded, 93 active+clean; 604 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 167/999 objects degraded (16.717%) 2024-09-17T09:31:04.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:04 smithi029 ceph-mon[58326]: pgmap v129: 161 pgs: 3 peering, 39 active+undersized, 26 active+undersized+degraded, 93 active+clean; 604 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 167/999 objects degraded (16.717%) 2024-09-17T09:31:04.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:04 smithi029 ceph-mon[60489]: pgmap v129: 161 pgs: 3 peering, 39 active+undersized, 26 active+undersized+degraded, 93 active+clean; 604 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 167/999 objects degraded (16.717%) 2024-09-17T09:31:05.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:05 smithi103 ceph-mon[49967]: pgmap v131: 161 pgs: 3 peering, 39 active+undersized, 26 active+undersized+degraded, 93 active+clean; 604 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 167/999 objects degraded (16.717%) 2024-09-17T09:31:05.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:05 smithi029 ceph-mon[58326]: pgmap v131: 161 pgs: 3 peering, 39 active+undersized, 26 active+undersized+degraded, 93 active+clean; 604 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 167/999 objects degraded (16.717%) 2024-09-17T09:31:05.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:05 smithi029 ceph-mon[60489]: pgmap v131: 161 pgs: 3 peering, 39 active+undersized, 26 active+undersized+degraded, 93 active+clean; 604 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 167/999 objects degraded (16.717%) 2024-09-17T09:31:06.048 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:31:05 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:31:05.728+0000 7fe73b7e7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-17T09:31:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:31:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:31:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:31:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: pgmap v132: 161 pgs: 3 peering, 26 active+undersized, 16 active+undersized+degraded, 116 active+clean; 604 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 91/999 objects degraded (9.109%) 2024-09-17T09:31:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: pgmap v133: 161 pgs: 3 peering, 26 active+undersized, 16 active+undersized+degraded, 116 active+clean; 604 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 91/999 objects degraded (9.109%) 2024-09-17T09:31:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:31:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:07.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:07.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T09:31:07.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T09:31:07.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:06 smithi103 ceph-mon[49967]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-17T09:31:07.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:07.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:07.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:31:07.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:31:07.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:31:07.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: pgmap v132: 161 pgs: 3 peering, 26 active+undersized, 16 active+undersized+degraded, 116 active+clean; 604 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 91/999 objects degraded (9.109%) 2024-09-17T09:31:07.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: pgmap v133: 161 pgs: 3 peering, 26 active+undersized, 16 active+undersized+degraded, 116 active+clean; 604 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 91/999 objects degraded (9.109%) 2024-09-17T09:31:07.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:07.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:31:07.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:07.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:07.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:07.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T09:31:07.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T09:31:07.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[58326]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-17T09:31:07.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:07.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:07.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:31:07.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:31:07.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:31:07.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: pgmap v132: 161 pgs: 3 peering, 26 active+undersized, 16 active+undersized+degraded, 116 active+clean; 604 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 91/999 objects degraded (9.109%) 2024-09-17T09:31:07.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: pgmap v133: 161 pgs: 3 peering, 26 active+undersized, 16 active+undersized+degraded, 116 active+clean; 604 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 91/999 objects degraded (9.109%) 2024-09-17T09:31:07.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:07.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:31:07.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:07.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:07.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:07.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T09:31:07.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T09:31:07.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:06 smithi029 ceph-mon[60489]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-17T09:31:08.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:07 smithi103 ceph-mon[49967]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 91/999 objects degraded (9.109%), 16 pgs degraded) 2024-09-17T09:31:08.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:07 smithi029 ceph-mon[58326]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 91/999 objects degraded (9.109%), 16 pgs degraded) 2024-09-17T09:31:08.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:07 smithi029 ceph-mon[60489]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 91/999 objects degraded (9.109%), 16 pgs degraded) 2024-09-17T09:31:09.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:08 smithi103 ceph-mon[49967]: pgmap v134: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:09.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:08 smithi029 ceph-mon[58326]: pgmap v134: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:09.030 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:08 smithi029 ceph-mon[60489]: pgmap v134: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:09.964 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:09 smithi029 ceph-mon[58326]: from='client.44724 172.21.15.29:0/661925301' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:31:09.964 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:09 smithi029 ceph-mon[60489]: from='client.44724 172.21.15.29:0/661925301' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:31:10.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:09 smithi103 ceph-mon[49967]: from='client.44724 172.21.15.29:0/661925301' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:31:11.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:10 smithi103 ceph-mon[49967]: pgmap v135: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:11.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:10 smithi103 ceph-mon[49967]: from='client.54613 172.21.15.29:0/125919941' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:31:11.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:10 smithi029 ceph-mon[58326]: pgmap v135: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:11.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:10 smithi029 ceph-mon[58326]: from='client.54613 172.21.15.29:0/125919941' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:31:11.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:10 smithi029 ceph-mon[60489]: pgmap v135: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:11.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:10 smithi029 ceph-mon[60489]: from='client.54613 172.21.15.29:0/125919941' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:31:13.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:12 smithi103 ceph-mon[49967]: pgmap v136: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:13.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:12 smithi029 ceph-mon[58326]: pgmap v136: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:13.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:12 smithi029 ceph-mon[60489]: pgmap v136: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:15.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:14 smithi103 ceph-mon[49967]: pgmap v137: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 110 B/s rd, 0 op/s 2024-09-17T09:31:15.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:14 smithi029 ceph-mon[58326]: pgmap v137: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 110 B/s rd, 0 op/s 2024-09-17T09:31:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:15.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:14 smithi029 ceph-mon[60489]: pgmap v137: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 110 B/s rd, 0 op/s 2024-09-17T09:31:15.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:17.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:16 smithi103 ceph-mon[49967]: pgmap v138: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T09:31:17.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:16 smithi029 ceph-mon[58326]: pgmap v138: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T09:31:17.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:16 smithi029 ceph-mon[60489]: pgmap v138: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-09-17T09:31:18.980 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:18 smithi029 ceph-mon[60489]: pgmap v139: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:31:18.981 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:18 smithi029 ceph-mon[58326]: pgmap v139: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:31:18.981 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:31:19.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:18 smithi103 ceph-mon[49967]: pgmap v139: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:31:19.646 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:31:19.646 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (4m) 36s ago 9m 19.1M - 0.20.0 0881eb8f169f 7bd158a6d65c 2024-09-17T09:31:19.646 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (4m) 17s ago 9m 36.7M - 6.7.4 557c83e11646 e361816f7c14 2024-09-17T09:31:19.646 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 running (38s) 36s ago 8m - - 24ecd6d5f14c 8d25644a0887 2024-09-17T09:31:19.646 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (3m) 17s ago 13m 437M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:31:19.646 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (4m) 36s ago 14m 485M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:31:19.647 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (3m) 36s ago 14m 49.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:31:19.647 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (2m) 17s ago 13m 40.9M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 da050285f124 2024-09-17T09:31:19.647 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (3m) 36s ago 13m 45.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 0d3d1a65e2c9 2024-09-17T09:31:19.647 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (5m) 36s ago 10m 15.0M - 0.18.1 e5a616e4b9cf 34bfa263169f 2024-09-17T09:31:19.647 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (4m) 17s ago 10m 8866k - 0.18.1 e5a616e4b9cf 1adf1b5baad1 2024-09-17T09:31:19.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (117s) 36s ago 13m 52.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 4c456435000b 2024-09-17T09:31:19.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (82s) 36s ago 12m 44.3M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2ff250ddb1d9 2024-09-17T09:31:19.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (61s) 36s ago 12m 49.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7899f3c3c2f4 2024-09-17T09:31:19.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (42s) 36s ago 12m 12.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 94b84aca13ce 2024-09-17T09:31:19.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (22s) 17s ago 11m 11.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2306402255ea 2024-09-17T09:31:19.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (11m) 17s ago 11m 52.4M 4096M 16.2.0 24ecd6d5f14c 10a64919fbc9 2024-09-17T09:31:19.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (11m) 17s ago 11m 50.1M 4096M 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:31:19.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (10m) 17s ago 10m 56.9M 4096M 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:31:19.648 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 error 17s ago 10m - - 2024-09-17T09:31:19.648 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (8m) 36s ago 8m 41.6M - 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:31:19.648 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (8m) 17s ago 8m 41.3M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:31:19.648 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (8m) 36s ago 8m 39.3M - 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:31:19.648 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (8m) 17s ago 8m 38.9M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:31:19.998 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:19 smithi103 ceph-mon[49967]: from='client.54619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:20.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:19 smithi029 ceph-mon[58326]: from='client.54619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:20.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:19 smithi029 ceph-mon[60489]: from='client.54619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:20.049 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:31:20.049 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:31:20.049 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:31:20.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:31:20.049 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:31:20.049 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:31:20.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:31:20.050 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:31:20.050 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 3, 2024-09-17T09:31:20.050 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 5 2024-09-17T09:31:20.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:31:20.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:31:20.050 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:31:20.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:31:20.050 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:31:20.050 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 7, 2024-09-17T09:31:20.050 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 10 2024-09-17T09:31:20.050 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:31:20.050 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:31:20.390 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:31:20.390 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:31:20.390 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:31:20.390 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:31:20.390 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [ 2024-09-17T09:31:20.390 INFO:teuthology.orchestra.run.smithi029.stdout: "mon", 2024-09-17T09:31:20.390 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr" 2024-09-17T09:31:20.390 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:31:20.391 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "10/23 daemons upgraded", 2024-09-17T09:31:20.391 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T09:31:20.391 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:31:20.391 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:31:20.793 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T09:31:20.793 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T09:31:20.794 INFO:teuthology.orchestra.run.smithi029.stdout: daemon prometheus.a on smithi103 is in error state 2024-09-17T09:31:21.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:20 smithi103 ceph-mon[49967]: from='client.44742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:21.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:20 smithi103 ceph-mon[49967]: pgmap v140: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:31:21.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:20 smithi103 ceph-mon[49967]: from='client.44748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:21.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:20 smithi103 ceph-mon[49967]: from='client.44754 172.21.15.29:0/20762203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:21.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:20 smithi029 ceph-mon[58326]: from='client.44742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:21.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:20 smithi029 ceph-mon[58326]: pgmap v140: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:31:21.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:20 smithi029 ceph-mon[58326]: from='client.44748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:21.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:20 smithi029 ceph-mon[58326]: from='client.44754 172.21.15.29:0/20762203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:21.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:20 smithi029 ceph-mon[60489]: from='client.44742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:21.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:20 smithi029 ceph-mon[60489]: pgmap v140: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:31:21.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:20 smithi029 ceph-mon[60489]: from='client.44748 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:21.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:20 smithi029 ceph-mon[60489]: from='client.44754 172.21.15.29:0/20762203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:21.869 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:21 smithi029 ceph-mon[60489]: from='client.44760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:21.869 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:21 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T09:31:21.869 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:21 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T09:31:21.869 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:21 smithi029 ceph-mon[60489]: Upgrade: osd.5 is safe to restart 2024-09-17T09:31:21.869 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:21 smithi029 ceph-mon[60489]: from='client.54649 172.21.15.29:0/2158831434' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:31:21.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:21 smithi029 ceph-mon[58326]: from='client.44760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:21.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:21 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T09:31:21.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:21 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T09:31:21.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:21 smithi029 ceph-mon[58326]: Upgrade: osd.5 is safe to restart 2024-09-17T09:31:21.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:21 smithi029 ceph-mon[58326]: from='client.54649 172.21.15.29:0/2158831434' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:31:22.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:21 smithi103 ceph-mon[49967]: from='client.44760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:22.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:21 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T09:31:22.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:21 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T09:31:22.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:21 smithi103 ceph-mon[49967]: Upgrade: osd.5 is safe to restart 2024-09-17T09:31:22.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:21 smithi103 ceph-mon[49967]: from='client.54649 172.21.15.29:0/2158831434' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:31:23.433 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[60489]: pgmap v141: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:31:23.433 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[60489]: Upgrade: Updating osd.5 2024-09-17T09:31:23.434 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:23.434 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T09:31:23.434 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:31:23.434 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[60489]: Deploying daemon osd.5 on smithi103 2024-09-17T09:31:23.434 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[60489]: from='client.54667 172.21.15.29:0/3455223529' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:31:23.434 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[58326]: pgmap v141: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:31:23.434 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[58326]: Upgrade: Updating osd.5 2024-09-17T09:31:23.434 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:23.434 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T09:31:23.434 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:31:23.435 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[58326]: Deploying daemon osd.5 on smithi103 2024-09-17T09:31:23.435 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:23 smithi029 ceph-mon[58326]: from='client.54667 172.21.15.29:0/3455223529' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:31:23.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:23 smithi103 ceph-mon[49967]: pgmap v141: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:31:23.490 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:23 smithi103 ceph-mon[49967]: Upgrade: Updating osd.5 2024-09-17T09:31:23.490 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:23 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:23.490 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:23 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T09:31:23.490 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:23 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:31:23.490 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:23 smithi103 ceph-mon[49967]: Deploying daemon osd.5 on smithi103 2024-09-17T09:31:23.490 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:23 smithi103 ceph-mon[49967]: from='client.54667 172.21.15.29:0/3455223529' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:31:24.455 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:24 smithi103 ceph-mon[49967]: from='client.44790 172.21.15.29:0/2343043985' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:31:24.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:24 smithi029 ceph-mon[58326]: from='client.44790 172.21.15.29:0/2343043985' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:31:24.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:24 smithi029 ceph-mon[60489]: from='client.44790 172.21.15.29:0/2343043985' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:31:25.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:25 smithi103 ceph-mon[49967]: pgmap v142: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:31:25.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:25 smithi103 systemd[1]: Stopping Ceph osd.5 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:31:25.507 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.5[28976]: debug 2024-09-17T09:31:25.283+0000 7fe025ea2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T09:31:25.507 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.5[28976]: debug 2024-09-17T09:31:25.283+0000 7fe025ea2700 -1 osd.5 103 *** Got signal Terminated *** 2024-09-17T09:31:25.507 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:25 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.5[28976]: debug 2024-09-17T09:31:25.283+0000 7fe025ea2700 -1 osd.5 103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:31:25.507 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:25 smithi103 podman[55429]: 2024-09-17 09:31:25.325092417 +0000 UTC m=+0.153043441 container died 10a64919fbc929ec30a7a420e43e2608b61191d4e08be83019169d28517aa519 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.name=CentOS Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Dimitri Savineau ) 2024-09-17T09:31:25.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:25 smithi029 ceph-mon[58326]: pgmap v142: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:31:25.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:25 smithi029 ceph-mon[60489]: pgmap v142: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-09-17T09:31:25.945 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:25 smithi103 podman[55429]: 2024-09-17 09:31:25.700501626 +0000 UTC m=+0.528452641 container cleanup 10a64919fbc929ec30a7a420e43e2608b61191d4e08be83019169d28517aa519 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.5, ceph=True, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, maintainer=Dimitri Savineau ) 2024-09-17T09:31:25.945 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:25 smithi103 bash[55429]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.5 2024-09-17T09:31:26.255 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:25 smithi103 podman[55440]: 2024-09-17 09:31:25.943746543 +0000 UTC m=+0.619685017 container remove 10a64919fbc929ec30a7a420e43e2608b61191d4e08be83019169d28517aa519 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.5, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, RELEASE=HEAD) 2024-09-17T09:31:26.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 failed (root=default,host=smithi103) (connection refused reported by osd.1) 2024-09-17T09:31:26.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.7 2024-09-17T09:31:26.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.6 2024-09-17T09:31:26.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.7 2024-09-17T09:31:26.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.4 2024-09-17T09:31:26.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.6 2024-09-17T09:31:26.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.3 2024-09-17T09:31:26.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.0 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.4 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.3 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.0 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.7 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.6 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.7 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.3 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.0 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.4 2024-09-17T09:31:26.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.6 2024-09-17T09:31:26.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.3 2024-09-17T09:31:26.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.0 2024-09-17T09:31:26.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.4 2024-09-17T09:31:26.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:26 smithi103 ceph-mon[49967]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.318 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.318 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 failed (root=default,host=smithi103) (connection refused reported by osd.1) 2024-09-17T09:31:26.318 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.7 2024-09-17T09:31:26.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.6 2024-09-17T09:31:26.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.7 2024-09-17T09:31:26.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.4 2024-09-17T09:31:26.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.6 2024-09-17T09:31:26.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.3 2024-09-17T09:31:26.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.0 2024-09-17T09:31:26.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.4 2024-09-17T09:31:26.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.3 2024-09-17T09:31:26.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.0 2024-09-17T09:31:26.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.7 2024-09-17T09:31:26.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.6 2024-09-17T09:31:26.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.7 2024-09-17T09:31:26.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.3 2024-09-17T09:31:26.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.0 2024-09-17T09:31:26.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.4 2024-09-17T09:31:26.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.6 2024-09-17T09:31:26.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.3 2024-09-17T09:31:26.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.0 2024-09-17T09:31:26.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.4 2024-09-17T09:31:26.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.321 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.321 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[60489]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 failed (root=default,host=smithi103) (connection refused reported by osd.1) 2024-09-17T09:31:26.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.7 2024-09-17T09:31:26.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.6 2024-09-17T09:31:26.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.7 2024-09-17T09:31:26.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.4 2024-09-17T09:31:26.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.6 2024-09-17T09:31:26.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.3 2024-09-17T09:31:26.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.0 2024-09-17T09:31:26.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.4 2024-09-17T09:31:26.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.3 2024-09-17T09:31:26.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.0 2024-09-17T09:31:26.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.7 2024-09-17T09:31:26.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.6 2024-09-17T09:31:26.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.7 2024-09-17T09:31:26.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.3 2024-09-17T09:31:26.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.0 2024-09-17T09:31:26.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.4 2024-09-17T09:31:26.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.6 2024-09-17T09:31:26.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.3 2024-09-17T09:31:26.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.0 2024-09-17T09:31:26.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.4 2024-09-17T09:31:26.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.2 2024-09-17T09:31:26.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:26 smithi029 ceph-mon[58326]: osd.5 reported immediately failed by osd.1 2024-09-17T09:31:26.722 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:26 smithi103 podman[55547]: 2024-09-17 09:31:26.324328023 +0000 UTC m=+0.015944833 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:31:26.723 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:26 smithi103 podman[55547]: 2024-09-17 09:31:26.452963445 +0000 UTC m=+0.144580247 container create b7680abf1e2179007fb49d8c7b3946484c7c52dce63ca00c0d8637c136725340 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T09:31:27.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:26 smithi103 podman[55547]: 2024-09-17 09:31:26.72212867 +0000 UTC m=+0.413745471 container init b7680abf1e2179007fb49d8c7b3946484c7c52dce63ca00c0d8637c136725340 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=reef-19e2a2b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:31:27.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:26 smithi103 podman[55547]: 2024-09-17 09:31:26.725613325 +0000 UTC m=+0.417230129 container start b7680abf1e2179007fb49d8c7b3946484c7c52dce63ca00c0d8637c136725340 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-17T09:31:27.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:26 smithi103 podman[55547]: 2024-09-17 09:31:26.805511454 +0000 UTC m=+0.497128266 container attach b7680abf1e2179007fb49d8c7b3946484c7c52dce63ca00c0d8637c136725340 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:31:27.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:26 smithi103 podman[55547]: 2024-09-17 09:31:26.908383854 +0000 UTC m=+0.600000656 container died b7680abf1e2179007fb49d8c7b3946484c7c52dce63ca00c0d8637c136725340 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:31:27.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:27 smithi103 ceph-mon[49967]: pgmap v143: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:27.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:27 smithi103 ceph-mon[49967]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:31:27.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:27 smithi103 ceph-mon[49967]: osdmap e104: 8 total, 7 up, 8 in 2024-09-17T09:31:27.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:27 smithi029 ceph-mon[58326]: pgmap v143: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:27.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:27 smithi029 ceph-mon[58326]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:31:27.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:27 smithi029 ceph-mon[58326]: osdmap e104: 8 total, 7 up, 8 in 2024-09-17T09:31:27.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:27 smithi029 ceph-mon[60489]: pgmap v143: 161 pgs: 161 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:27.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:27 smithi029 ceph-mon[60489]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:31:27.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:27 smithi029 ceph-mon[60489]: osdmap e104: 8 total, 7 up, 8 in 2024-09-17T09:31:27.978 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:27 smithi103 podman[55570]: 2024-09-17 09:31:27.542836573 +0000 UTC m=+0.626488565 container remove b7680abf1e2179007fb49d8c7b3946484c7c52dce63ca00c0d8637c136725340 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:31:27.978 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:27 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.5.service: Deactivated successfully. 2024-09-17T09:31:27.978 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:27 smithi103 systemd[1]: Stopped Ceph osd.5 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:31:27.978 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:27 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.5.service: Consumed 1.114s CPU time. 2024-09-17T09:31:27.978 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:27 smithi103 systemd[1]: Starting Ceph osd.5 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:31:28.239 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:28 smithi103 ceph-mon[49967]: osdmap e105: 8 total, 7 up, 8 in 2024-09-17T09:31:28.239 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:27 smithi103 podman[55671]: 2024-09-17 09:31:27.879532252 +0000 UTC m=+0.016172684 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:31:28.239 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 podman[55671]: 2024-09-17 09:31:28.010822659 +0000 UTC m=+0.147463118 container create da5a8262fee856b2cab1a59105792de84215c41ae86fac9ed4ca5470d234dcef (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-17T09:31:28.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 podman[55671]: 2024-09-17 09:31:28.238260814 +0000 UTC m=+0.374901240 container init da5a8262fee856b2cab1a59105792de84215c41ae86fac9ed4ca5470d234dcef (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-17T09:31:28.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 podman[55671]: 2024-09-17 09:31:28.241609854 +0000 UTC m=+0.378250280 container start da5a8262fee856b2cab1a59105792de84215c41ae86fac9ed4ca5470d234dcef (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-activate, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=reef-19e2a2b) 2024-09-17T09:31:28.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 podman[55671]: 2024-09-17 09:31:28.330968207 +0000 UTC m=+0.467608625 container attach da5a8262fee856b2cab1a59105792de84215c41ae86fac9ed4ca5470d234dcef (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:31:28.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:28 smithi029 ceph-mon[58326]: osdmap e105: 8 total, 7 up, 8 in 2024-09-17T09:31:28.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:28 smithi029 ceph-mon[60489]: osdmap e105: 8 total, 7 up, 8 in 2024-09-17T09:31:29.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-activate[55686]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T09:31:29.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 bash[55671]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T09:31:29.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-activate[55686]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T09:31:29.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 bash[55671]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T09:31:29.007 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-activate[55686]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T09:31:29.007 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 bash[55671]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T09:31:29.007 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-activate[55686]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T09:31:29.007 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 bash[55671]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T09:31:29.007 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-activate[55686]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-17T09:31:29.007 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 bash[55671]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-17T09:31:29.007 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-activate[55686]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T09:31:29.007 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 bash[55671]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T09:31:29.007 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-activate[55686]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-17T09:31:29.007 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 bash[55671]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-17T09:31:29.007 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:28 smithi103 podman[55671]: 2024-09-17 09:31:28.582777313 +0000 UTC m=+0.719417741 container died da5a8262fee856b2cab1a59105792de84215c41ae86fac9ed4ca5470d234dcef (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-17T09:31:29.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:29 smithi103 ceph-mon[49967]: pgmap v146: 161 pgs: 22 peering, 14 stale+active+clean, 125 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:29.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:29 smithi103 ceph-mon[49967]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-17T09:31:29.413 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:29 smithi103 podman[55804]: 2024-09-17 09:31:29.205359768 +0000 UTC m=+0.617113995 container remove da5a8262fee856b2cab1a59105792de84215c41ae86fac9ed4ca5470d234dcef (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b) 2024-09-17T09:31:29.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:29 smithi029 ceph-mon[58326]: pgmap v146: 161 pgs: 22 peering, 14 stale+active+clean, 125 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:29.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:29 smithi029 ceph-mon[58326]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-17T09:31:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:29 smithi029 ceph-mon[60489]: pgmap v146: 161 pgs: 22 peering, 14 stale+active+clean, 125 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:29 smithi029 ceph-mon[60489]: Health check failed: Reduced data availability: 2 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-17T09:31:29.703 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:29 smithi103 podman[55867]: 2024-09-17 09:31:29.314155934 +0000 UTC m=+0.015986747 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:31:29.703 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:29 smithi103 podman[55867]: 2024-09-17 09:31:29.435274873 +0000 UTC m=+0.137105728 container create 0312d071d90be31f4e4585ca386d3d447e80601ec97d6661d27a00de7897d403 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD) 2024-09-17T09:31:30.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:29 smithi103 podman[55867]: 2024-09-17 09:31:29.702330334 +0000 UTC m=+0.404161157 container init 0312d071d90be31f4e4585ca386d3d447e80601ec97d6661d27a00de7897d403 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-17T09:31:30.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:29 smithi103 podman[55867]: 2024-09-17 09:31:29.705901094 +0000 UTC m=+0.407731910 container start 0312d071d90be31f4e4585ca386d3d447e80601ec97d6661d27a00de7897d403 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b) 2024-09-17T09:31:30.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:29 smithi103 ceph-osd[55887]: -- 172.21.15.103:0/409899224 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5608566df1e0 con 0x56085745e000 2024-09-17T09:31:30.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:29 smithi103 bash[55867]: 0312d071d90be31f4e4585ca386d3d447e80601ec97d6661d27a00de7897d403 2024-09-17T09:31:30.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:29 smithi103 systemd[1]: Started Ceph osd.5 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:31:30.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:31:30.245+0000 7f56e6ea1640 -1 Falling back to public interface 2024-09-17T09:31:31.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:30 smithi103 ceph-mon[49967]: pgmap v147: 161 pgs: 22 peering, 14 stale+active+clean, 125 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:31.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:30 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:31.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:30 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:31.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:30 smithi029 ceph-mon[58326]: pgmap v147: 161 pgs: 22 peering, 14 stale+active+clean, 125 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:31.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:30 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:31.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:30 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:31.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:30 smithi029 ceph-mon[60489]: pgmap v147: 161 pgs: 22 peering, 14 stale+active+clean, 125 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:31.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:30 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:31.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:30 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:32.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:31 smithi103 ceph-mon[49967]: Health check failed: Degraded data redundancy: 83/999 objects degraded (8.308%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T09:31:32.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:31 smithi029 ceph-mon[58326]: Health check failed: Degraded data redundancy: 83/999 objects degraded (8.308%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T09:31:32.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:31 smithi029 ceph-mon[60489]: Health check failed: Degraded data redundancy: 83/999 objects degraded (8.308%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T09:31:33.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:32 smithi103 ceph-mon[49967]: pgmap v148: 161 pgs: 30 active+undersized, 16 peering, 13 active+undersized+degraded, 102 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 83/999 objects degraded (8.308%) 2024-09-17T09:31:33.256 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:31:32.944+0000 7f56e6ea1640 -1 osd.5 0 read_superblock omap replica is missing. 2024-09-17T09:31:33.256 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:32 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:31:32.967+0000 7f56e6ea1640 -1 osd.5 103 log_to_monitors true 2024-09-17T09:31:33.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:32 smithi029 ceph-mon[58326]: pgmap v148: 161 pgs: 30 active+undersized, 16 peering, 13 active+undersized+degraded, 102 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 83/999 objects degraded (8.308%) 2024-09-17T09:31:33.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:32 smithi029 ceph-mon[60489]: pgmap v148: 161 pgs: 30 active+undersized, 16 peering, 13 active+undersized+degraded, 102 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 83/999 objects degraded (8.308%) 2024-09-17T09:31:34.184 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:33 smithi103 ceph-mon[49967]: from='osd.5 [v2:172.21.15.103:6808/3665332505,v1:172.21.15.103:6809/3665332505]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:31:34.185 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:33 smithi103 ceph-mon[49967]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:31:34.185 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:31:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:31:33.923+0000 7f56dec4b640 -1 osd.5 103 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:31:34.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:33 smithi029 ceph-mon[58326]: from='osd.5 [v2:172.21.15.103:6808/3665332505,v1:172.21.15.103:6809/3665332505]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:31:34.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:33 smithi029 ceph-mon[58326]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:31:34.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:33 smithi029 ceph-mon[60489]: from='osd.5 [v2:172.21.15.103:6808/3665332505,v1:172.21.15.103:6809/3665332505]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:31:34.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:33 smithi029 ceph-mon[60489]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T09:31:35.149 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[60489]: pgmap v149: 161 pgs: 41 active+undersized, 18 active+undersized+degraded, 102 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 113/999 objects degraded (11.311%) 2024-09-17T09:31:35.150 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[60489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-09-17T09:31:35.150 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[60489]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T09:31:35.150 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[60489]: osdmap e106: 8 total, 7 up, 8 in 2024-09-17T09:31:35.150 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[60489]: from='osd.5 [v2:172.21.15.103:6808/3665332505,v1:172.21.15.103:6809/3665332505]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:31:35.150 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[60489]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:31:35.150 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:35.150 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:35.150 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:35.150 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:35.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[58326]: pgmap v149: 161 pgs: 41 active+undersized, 18 active+undersized+degraded, 102 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 113/999 objects degraded (11.311%) 2024-09-17T09:31:35.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[58326]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-09-17T09:31:35.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[58326]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T09:31:35.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[58326]: osdmap e106: 8 total, 7 up, 8 in 2024-09-17T09:31:35.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[58326]: from='osd.5 [v2:172.21.15.103:6808/3665332505,v1:172.21.15.103:6809/3665332505]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:31:35.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[58326]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:31:35.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:35.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:35.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:35.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:34 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:35.174 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:34 smithi103 ceph-mon[49967]: pgmap v149: 161 pgs: 41 active+undersized, 18 active+undersized+degraded, 102 active+clean; 604 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 113/999 objects degraded (11.311%) 2024-09-17T09:31:35.174 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:34 smithi103 ceph-mon[49967]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-09-17T09:31:35.174 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:34 smithi103 ceph-mon[49967]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T09:31:35.174 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:34 smithi103 ceph-mon[49967]: osdmap e106: 8 total, 7 up, 8 in 2024-09-17T09:31:35.174 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:34 smithi103 ceph-mon[49967]: from='osd.5 [v2:172.21.15.103:6808/3665332505,v1:172.21.15.103:6809/3665332505]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:31:35.174 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:34 smithi103 ceph-mon[49967]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:31:35.174 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:34 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:35.174 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:34 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:35.175 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:34 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:35.175 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:34 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:36.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:35 smithi103 ceph-mon[49967]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:31:36.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:35 smithi103 ceph-mon[49967]: osd.5 [v2:172.21.15.103:6808/3665332505,v1:172.21.15.103:6809/3665332505] boot 2024-09-17T09:31:36.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:35 smithi103 ceph-mon[49967]: osdmap e107: 8 total, 8 up, 8 in 2024-09-17T09:31:36.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:35 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:31:36.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:35 smithi103 ceph-mon[49967]: osdmap e108: 8 total, 8 up, 8 in 2024-09-17T09:31:36.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:35 smithi029 ceph-mon[58326]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:31:36.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:35 smithi029 ceph-mon[58326]: osd.5 [v2:172.21.15.103:6808/3665332505,v1:172.21.15.103:6809/3665332505] boot 2024-09-17T09:31:36.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:35 smithi029 ceph-mon[58326]: osdmap e107: 8 total, 8 up, 8 in 2024-09-17T09:31:36.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:35 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:31:36.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:35 smithi029 ceph-mon[58326]: osdmap e108: 8 total, 8 up, 8 in 2024-09-17T09:31:36.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:35 smithi029 ceph-mon[60489]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:31:36.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:35 smithi029 ceph-mon[60489]: osd.5 [v2:172.21.15.103:6808/3665332505,v1:172.21.15.103:6809/3665332505] boot 2024-09-17T09:31:36.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:35 smithi029 ceph-mon[60489]: osdmap e107: 8 total, 8 up, 8 in 2024-09-17T09:31:36.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:35 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T09:31:36.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:35 smithi029 ceph-mon[60489]: osdmap e108: 8 total, 8 up, 8 in 2024-09-17T09:31:37.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:36 smithi103 ceph-mon[49967]: pgmap v152: 161 pgs: 10 peering, 35 active+undersized, 14 active+undersized+degraded, 102 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 89/999 objects degraded (8.909%) 2024-09-17T09:31:37.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:36 smithi103 ceph-mon[49967]: from='client.54694 172.21.15.29:0/3008143502' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:31:37.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:36 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/4287044466' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:31:37.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:36 smithi029 ceph-mon[58326]: pgmap v152: 161 pgs: 10 peering, 35 active+undersized, 14 active+undersized+degraded, 102 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 89/999 objects degraded (8.909%) 2024-09-17T09:31:37.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:36 smithi029 ceph-mon[58326]: from='client.54694 172.21.15.29:0/3008143502' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:31:37.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:36 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/4287044466' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:31:37.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:36 smithi029 ceph-mon[60489]: pgmap v152: 161 pgs: 10 peering, 35 active+undersized, 14 active+undersized+degraded, 102 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 89/999 objects degraded (8.909%) 2024-09-17T09:31:37.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:36 smithi029 ceph-mon[60489]: from='client.54694 172.21.15.29:0/3008143502' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist ls"}]: dispatch 2024-09-17T09:31:37.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:36 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/4287044466' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blacklist", "blacklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: access denied 2024-09-17T09:31:38.210 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:37 smithi029 ceph-mon[58326]: Health check update: Degraded data redundancy: 89/999 objects degraded (8.909%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T09:31:38.210 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:37 smithi029 ceph-mon[60489]: Health check update: Degraded data redundancy: 89/999 objects degraded (8.909%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T09:31:38.227 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:37 smithi103 ceph-mon[49967]: Health check update: Degraded data redundancy: 89/999 objects degraded (8.909%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T09:31:38.548 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:31:38.403+0000 7fe73b7e7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T09:31:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:38 smithi103 ceph-mon[49967]: pgmap v154: 161 pgs: 10 peering, 14 active+undersized, 7 active+undersized+degraded, 130 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 47/999 objects degraded (4.705%) 2024-09-17T09:31:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:31:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:31:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:31:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:39.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T09:31:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[60489]: pgmap v154: 161 pgs: 10 peering, 14 active+undersized, 7 active+undersized+degraded, 130 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 47/999 objects degraded (4.705%) 2024-09-17T09:31:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:31:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:31:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:39.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:31:39.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:39.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:39.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:39.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T09:31:39.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[58326]: pgmap v154: 161 pgs: 10 peering, 14 active+undersized, 7 active+undersized+degraded, 130 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 47/999 objects degraded (4.705%) 2024-09-17T09:31:39.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:39.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:39.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:31:39.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:31:39.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:39.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:31:39.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:39.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:39.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:39.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T09:31:40.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:39 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:31:40.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:39 smithi103 ceph-mon[49967]: pgmap v155: 161 pgs: 10 peering, 14 active+undersized, 7 active+undersized+degraded, 130 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 219 B/s rd, 0 op/s; 47/999 objects degraded (4.705%) 2024-09-17T09:31:40.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:39 smithi103 ceph-mon[49967]: pgmap v156: 161 pgs: 10 peering, 14 active+undersized, 7 active+undersized+degraded, 130 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 234 B/s rd, 0 op/s; 47/999 objects degraded (4.705%) 2024-09-17T09:31:40.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:39 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T09:31:40.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:39 smithi103 ceph-mon[49967]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T09:31:40.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:39 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:40.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:39 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:31:40.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:39 smithi029 ceph-mon[58326]: pgmap v155: 161 pgs: 10 peering, 14 active+undersized, 7 active+undersized+degraded, 130 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 219 B/s rd, 0 op/s; 47/999 objects degraded (4.705%) 2024-09-17T09:31:40.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:39 smithi029 ceph-mon[58326]: pgmap v156: 161 pgs: 10 peering, 14 active+undersized, 7 active+undersized+degraded, 130 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 234 B/s rd, 0 op/s; 47/999 objects degraded (4.705%) 2024-09-17T09:31:40.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:39 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T09:31:40.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:39 smithi029 ceph-mon[58326]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T09:31:40.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:39 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:40.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:39 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:31:40.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:39 smithi029 ceph-mon[60489]: pgmap v155: 161 pgs: 10 peering, 14 active+undersized, 7 active+undersized+degraded, 130 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 219 B/s rd, 0 op/s; 47/999 objects degraded (4.705%) 2024-09-17T09:31:40.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:39 smithi029 ceph-mon[60489]: pgmap v156: 161 pgs: 10 peering, 14 active+undersized, 7 active+undersized+degraded, 130 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 234 B/s rd, 0 op/s; 47/999 objects degraded (4.705%) 2024-09-17T09:31:40.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:39 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T09:31:40.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:39 smithi029 ceph-mon[60489]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T09:31:40.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:39 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:41.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:40 smithi103 ceph-mon[49967]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 47/999 objects degraded (4.705%), 7 pgs degraded) 2024-09-17T09:31:41.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:40 smithi029 ceph-mon[58326]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 47/999 objects degraded (4.705%), 7 pgs degraded) 2024-09-17T09:31:41.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:40 smithi029 ceph-mon[60489]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 47/999 objects degraded (4.705%), 7 pgs degraded) 2024-09-17T09:31:42.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:41 smithi103 ceph-mon[49967]: pgmap v157: 161 pgs: 10 peering, 151 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 190 B/s rd, 0 op/s 2024-09-17T09:31:42.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:41 smithi029 ceph-mon[58326]: pgmap v157: 161 pgs: 10 peering, 151 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 190 B/s rd, 0 op/s 2024-09-17T09:31:42.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:41 smithi029 ceph-mon[60489]: pgmap v157: 161 pgs: 10 peering, 151 active+clean; 604 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 190 B/s rd, 0 op/s 2024-09-17T09:31:44.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:43 smithi103 ceph-mon[49967]: pgmap v158: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 153 B/s rd, 0 op/s 2024-09-17T09:31:44.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:43 smithi029 ceph-mon[58326]: pgmap v158: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 153 B/s rd, 0 op/s 2024-09-17T09:31:44.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:43 smithi029 ceph-mon[60489]: pgmap v158: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 153 B/s rd, 0 op/s 2024-09-17T09:31:46.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:45 smithi103 ceph-mon[49967]: pgmap v159: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2024-09-17T09:31:46.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:45 smithi029 ceph-mon[58326]: pgmap v159: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2024-09-17T09:31:46.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:45 smithi029 ceph-mon[60489]: pgmap v159: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 122 B/s rd, 0 op/s 2024-09-17T09:31:48.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:48 smithi103 ceph-mon[49967]: pgmap v160: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:48.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:48 smithi029 ceph-mon[58326]: pgmap v160: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:48.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:48 smithi029 ceph-mon[60489]: pgmap v160: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:50.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:50 smithi103 ceph-mon[49967]: pgmap v161: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:50.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:50 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:50 smithi029 ceph-mon[58326]: pgmap v161: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:50 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:50.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:50 smithi029 ceph-mon[60489]: pgmap v161: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:50.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:50 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:51.164 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:31:51.827 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:31:51.827 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (5m) 68s ago 10m 19.1M - 0.20.0 0881eb8f169f 7bd158a6d65c 2024-09-17T09:31:51.827 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (4m) 17s ago 9m 36.8M - 6.7.4 557c83e11646 e361816f7c14 2024-09-17T09:31:51.827 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 running (70s) 68s ago 9m - - 24ecd6d5f14c 8d25644a0887 2024-09-17T09:31:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (4m) 17s ago 13m 437M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:31:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (4m) 68s ago 15m 485M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:31:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (3m) 68s ago 15m 49.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:31:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (3m) 17s ago 14m 41.6M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 da050285f124 2024-09-17T09:31:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (3m) 68s ago 14m 45.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 0d3d1a65e2c9 2024-09-17T09:31:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (5m) 68s ago 10m 15.0M - 0.18.1 e5a616e4b9cf 34bfa263169f 2024-09-17T09:31:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (5m) 17s ago 10m 8866k - 0.18.1 e5a616e4b9cf 1adf1b5baad1 2024-09-17T09:31:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (2m) 68s ago 13m 52.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 4c456435000b 2024-09-17T09:31:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (115s) 68s ago 13m 44.3M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2ff250ddb1d9 2024-09-17T09:31:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (93s) 68s ago 13m 49.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7899f3c3c2f4 2024-09-17T09:31:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (74s) 68s ago 12m 12.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 94b84aca13ce 2024-09-17T09:31:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (54s) 17s ago 12m 43.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2306402255ea 2024-09-17T09:31:51.828 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (22s) 17s ago 12m 11.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0312d071d90b 2024-09-17T09:31:51.829 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (11m) 17s ago 11m 51.7M 4096M 16.2.0 24ecd6d5f14c c627177969a3 2024-09-17T09:31:51.829 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (11m) 17s ago 11m 57.2M 4096M 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:31:51.829 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 error 17s ago 10m - - 2024-09-17T09:31:51.829 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (9m) 68s ago 9m 41.6M - 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:31:51.829 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (9m) 17s ago 9m 41.5M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:31:51.829 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (9m) 68s ago 9m 39.3M - 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:31:51.829 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (9m) 17s ago 9m 39.1M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:31:52.235 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:31:52.235 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:31:52.235 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:31:52.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:31:52.235 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:31:52.236 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:31:52.236 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:31:52.236 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:31:52.236 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2, 2024-09-17T09:31:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 6 2024-09-17T09:31:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:31:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:31:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:31:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:31:52.237 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:31:52.238 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 6, 2024-09-17T09:31:52.238 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 11 2024-09-17T09:31:52.238 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:31:52.238 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:31:52.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:52 smithi029 ceph-mon[58326]: pgmap v162: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:52.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:52 smithi029 ceph-mon[58326]: from='client.44826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:52.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:52 smithi029 ceph-mon[60489]: pgmap v162: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:52.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:52 smithi029 ceph-mon[60489]: from='client.44826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:52.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:52 smithi103 ceph-mon[49967]: pgmap v162: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:52.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:52 smithi103 ceph-mon[49967]: from='client.44826 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:52.574 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:31:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:31:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:31:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:31:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [ 2024-09-17T09:31:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "mon", 2024-09-17T09:31:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr" 2024-09-17T09:31:52.574 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:31:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "11/23 daemons upgraded", 2024-09-17T09:31:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T09:31:52.575 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:31:52.575 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:31:52.975 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T09:31:52.975 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T09:31:52.975 INFO:teuthology.orchestra.run.smithi029.stdout: daemon prometheus.a on smithi103 is in error state 2024-09-17T09:31:53.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:53 smithi029 ceph-mon[60489]: from='client.54712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:53.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:53 smithi029 ceph-mon[60489]: from='client.54718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:53.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:53 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/3165546904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:53.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:53 smithi029 ceph-mon[60489]: from='client.54736 172.21.15.29:0/296360908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:31:53.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:53 smithi029 ceph-mon[58326]: from='client.54712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:53.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:53 smithi029 ceph-mon[58326]: from='client.54718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:53.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:53 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/3165546904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:53.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:53 smithi029 ceph-mon[58326]: from='client.54736 172.21.15.29:0/296360908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:31:53.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:53 smithi103 ceph-mon[49967]: from='client.54712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:53.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:53 smithi103 ceph-mon[49967]: from='client.54718 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:53.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:53 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/3165546904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:31:53.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:53 smithi103 ceph-mon[49967]: from='client.54736 172.21.15.29:0/296360908' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:31:54.011 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:54 smithi103 ceph-mon[49967]: pgmap v163: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:54.012 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:54 smithi103 ceph-mon[49967]: from='client.44850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:54.012 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:54 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T09:31:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:54 smithi029 ceph-mon[58326]: pgmap v163: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:54 smithi029 ceph-mon[58326]: from='client.44850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:54.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:54 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T09:31:54.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:54 smithi029 ceph-mon[60489]: pgmap v163: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:54.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:54 smithi029 ceph-mon[60489]: from='client.44850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:31:54.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:54 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T09:31:55.057 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:55 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T09:31:55.057 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:55 smithi103 ceph-mon[49967]: Upgrade: osd.6 is safe to restart 2024-09-17T09:31:55.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:55 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T09:31:55.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:55 smithi029 ceph-mon[58326]: Upgrade: osd.6 is safe to restart 2024-09-17T09:31:55.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:55 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T09:31:55.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:55 smithi029 ceph-mon[60489]: Upgrade: osd.6 is safe to restart 2024-09-17T09:31:56.379 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:56 smithi103 ceph-mon[49967]: pgmap v164: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:56.380 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:56 smithi103 ceph-mon[49967]: Upgrade: Updating osd.6 2024-09-17T09:31:56.380 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:56 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:56.380 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:56 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T09:31:56.380 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:56 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:31:56.380 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:56 smithi103 ceph-mon[49967]: Deploying daemon osd.6 on smithi103 2024-09-17T09:31:56.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:56 smithi029 ceph-mon[58326]: pgmap v164: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:56.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:56 smithi029 ceph-mon[58326]: Upgrade: Updating osd.6 2024-09-17T09:31:56.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:56 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:56.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:56 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T09:31:56.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:56 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:31:56.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:56 smithi029 ceph-mon[58326]: Deploying daemon osd.6 on smithi103 2024-09-17T09:31:56.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:56 smithi029 ceph-mon[60489]: pgmap v164: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:56.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:56 smithi029 ceph-mon[60489]: Upgrade: Updating osd.6 2024-09-17T09:31:56.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:56 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:31:56.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:56 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T09:31:56.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:56 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:31:56.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:56 smithi029 ceph-mon[60489]: Deploying daemon osd.6 on smithi103 2024-09-17T09:31:58.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:58 smithi103 ceph-mon[49967]: pgmap v165: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:58.257 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:57 smithi103 systemd[1]: Stopping Ceph osd.6 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:31:58.257 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.6[31353]: debug 2024-09-17T09:31:58.142+0000 7fe29fc22700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T09:31:58.257 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.6[31353]: debug 2024-09-17T09:31:58.142+0000 7fe29fc22700 -1 osd.6 108 *** Got signal Terminated *** 2024-09-17T09:31:58.257 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.6[31353]: debug 2024-09-17T09:31:58.142+0000 7fe29fc22700 -1 osd.6 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:31:58.257 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:58 smithi103 podman[57772]: 2024-09-17 09:31:58.173135699 +0000 UTC m=+0.136043134 container died c627177969a3db81dc1f88543c7fe545307c1931f67b59fbb2d1767abe9ce723 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.6, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Base Image, ceph=True, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, org.label-schema.build-date=20201204, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-17T09:31:58.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:58 smithi029 ceph-mon[58326]: pgmap v165: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:58.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:58 smithi029 ceph-mon[60489]: pgmap v165: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:31:58.736 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:58 smithi103 podman[57772]: 2024-09-17 09:31:58.567882952 +0000 UTC m=+0.530790386 container cleanup c627177969a3db81dc1f88543c7fe545307c1931f67b59fbb2d1767abe9ce723 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.6, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-16.2.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0) 2024-09-17T09:31:58.737 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:58 smithi103 bash[57772]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.6 2024-09-17T09:31:59.005 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:58 smithi103 podman[57785]: 2024-09-17 09:31:58.736007417 +0000 UTC m=+0.559724233 container remove c627177969a3db81dc1f88543c7fe545307c1931f67b59fbb2d1767abe9ce723 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.6, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20201204, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-16.2.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557) 2024-09-17T09:31:59.409 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:59 smithi103 podman[57892]: 2024-09-17 09:31:59.191126059 +0000 UTC m=+0.015859380 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:31:59.409 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:59 smithi103 podman[57892]: 2024-09-17 09:31:59.303734447 +0000 UTC m=+0.128467800 container create 995b11314aff3719405800a5866166ac003dfda0a50419b68e48704d7f78237b (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-17T09:31:59.410 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.3 2024-09-17T09:31:59.410 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 failed (root=default,host=smithi103) (connection refused reported by osd.3) 2024-09-17T09:31:59.410 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.3 2024-09-17T09:31:59.410 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.0 2024-09-17T09:31:59.410 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.0 2024-09-17T09:31:59.410 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.410 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.2 2024-09-17T09:31:59.410 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.410 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.410 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.2 2024-09-17T09:31:59.410 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.411 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.411 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.411 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.5 2024-09-17T09:31:59.411 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.5 2024-09-17T09:31:59.411 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.411 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.411 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.3 2024-09-17T09:31:59.411 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.3 2024-09-17T09:31:59.411 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.2 2024-09-17T09:31:59.411 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.411 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.2 2024-09-17T09:31:59.411 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.5 2024-09-17T09:31:59.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.0 2024-09-17T09:31:59.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.0 2024-09-17T09:31:59.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.5 2024-09-17T09:31:59.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:31:59 smithi103 ceph-mon[49967]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.3 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 failed (root=default,host=smithi103) (connection refused reported by osd.3) 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.3 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.0 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.0 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.2 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.2 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.5 2024-09-17T09:31:59.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.5 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.3 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.3 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.2 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.2 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.5 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.0 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.0 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.5 2024-09-17T09:31:59.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[58326]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.3 2024-09-17T09:31:59.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 failed (root=default,host=smithi103) (connection refused reported by osd.3) 2024-09-17T09:31:59.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.3 2024-09-17T09:31:59.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.0 2024-09-17T09:31:59.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.0 2024-09-17T09:31:59.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.2 2024-09-17T09:31:59.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.2 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.5 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.5 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.3 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.3 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.2 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.2 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.5 2024-09-17T09:31:59.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.0 2024-09-17T09:31:59.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.0 2024-09-17T09:31:59.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.5 2024-09-17T09:31:59.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.7 2024-09-17T09:31:59.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.4 2024-09-17T09:31:59.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:31:59 smithi029 ceph-mon[60489]: osd.6 reported immediately failed by osd.1 2024-09-17T09:31:59.712 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:59 smithi103 podman[57892]: 2024-09-17 09:31:59.523410527 +0000 UTC m=+0.348143844 container init 995b11314aff3719405800a5866166ac003dfda0a50419b68e48704d7f78237b (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-17T09:31:59.712 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:59 smithi103 podman[57892]: 2024-09-17 09:31:59.526725883 +0000 UTC m=+0.351459198 container start 995b11314aff3719405800a5866166ac003dfda0a50419b68e48704d7f78237b (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:31:59.712 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:59 smithi103 podman[57892]: 2024-09-17 09:31:59.623543968 +0000 UTC m=+0.448277290 container attach 995b11314aff3719405800a5866166ac003dfda0a50419b68e48704d7f78237b (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T09:32:00.005 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:31:59 smithi103 podman[57892]: 2024-09-17 09:31:59.711626714 +0000 UTC m=+0.536360040 container died 995b11314aff3719405800a5866166ac003dfda0a50419b68e48704d7f78237b (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-17T09:32:00.404 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:00 smithi103 ceph-mon[49967]: pgmap v166: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:32:00.404 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:00 smithi103 ceph-mon[49967]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:32:00.404 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:00 smithi103 ceph-mon[49967]: osdmap e109: 8 total, 7 up, 8 in 2024-09-17T09:32:00.404 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 6]}]: dispatch 2024-09-17T09:32:00.404 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 6]}]: dispatch 2024-09-17T09:32:00.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 0]}]: dispatch 2024-09-17T09:32:00.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-09-17T09:32:00.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 6]}]: dispatch 2024-09-17T09:32:00.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]: dispatch 2024-09-17T09:32:00.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 6]}]: dispatch 2024-09-17T09:32:00.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2, 4, 5]}]: dispatch 2024-09-17T09:32:00.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]: dispatch 2024-09-17T09:32:00.405 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 6]}]: dispatch 2024-09-17T09:32:00.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[58326]: pgmap v166: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:32:00.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[58326]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:32:00.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[58326]: osdmap e109: 8 total, 7 up, 8 in 2024-09-17T09:32:00.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 6]}]: dispatch 2024-09-17T09:32:00.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 6]}]: dispatch 2024-09-17T09:32:00.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 0]}]: dispatch 2024-09-17T09:32:00.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-09-17T09:32:00.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 6]}]: dispatch 2024-09-17T09:32:00.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]: dispatch 2024-09-17T09:32:00.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 6]}]: dispatch 2024-09-17T09:32:00.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2, 4, 5]}]: dispatch 2024-09-17T09:32:00.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]: dispatch 2024-09-17T09:32:00.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 6]}]: dispatch 2024-09-17T09:32:00.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[60489]: pgmap v166: 161 pgs: 161 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:32:00.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[60489]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:32:00.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[60489]: osdmap e109: 8 total, 7 up, 8 in 2024-09-17T09:32:00.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 6]}]: dispatch 2024-09-17T09:32:00.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 6]}]: dispatch 2024-09-17T09:32:00.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 0]}]: dispatch 2024-09-17T09:32:00.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-09-17T09:32:00.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 6]}]: dispatch 2024-09-17T09:32:00.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]: dispatch 2024-09-17T09:32:00.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 6]}]: dispatch 2024-09-17T09:32:00.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2, 4, 5]}]: dispatch 2024-09-17T09:32:00.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]: dispatch 2024-09-17T09:32:00.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 6]}]: dispatch 2024-09-17T09:32:00.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:00 smithi103 podman[57915]: 2024-09-17 09:32:00.40324362 +0000 UTC m=+0.685730882 container remove 995b11314aff3719405800a5866166ac003dfda0a50419b68e48704d7f78237b (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, ceph=True) 2024-09-17T09:32:00.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:00 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.6.service: Deactivated successfully. 2024-09-17T09:32:00.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:00 smithi103 systemd[1]: Stopped Ceph osd.6 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:32:00.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:00 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.6.service: Consumed 1.107s CPU time. 2024-09-17T09:32:00.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:00 smithi103 systemd[1]: Starting Ceph osd.6 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:32:01.104 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:00 smithi103 podman[58013]: 2024-09-17 09:32:00.744942955 +0000 UTC m=+0.016351577 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:32:01.104 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:00 smithi103 podman[58013]: 2024-09-17 09:32:00.871325668 +0000 UTC m=+0.142734290 container create c9a6e1ebb1927380cca0d08635a5996af6eb8919b62327d25be763b3a7c3bb8b (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-17T09:32:01.432 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 podman[58013]: 2024-09-17 09:32:01.132303101 +0000 UTC m=+0.403711722 container init c9a6e1ebb1927380cca0d08635a5996af6eb8919b62327d25be763b3a7c3bb8b (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:32:01.432 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 podman[58013]: 2024-09-17 09:32:01.135763341 +0000 UTC m=+0.407171963 container start c9a6e1ebb1927380cca0d08635a5996af6eb8919b62327d25be763b3a7c3bb8b (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-activate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-17T09:32:01.432 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 podman[58013]: 2024-09-17 09:32:01.232538146 +0000 UTC m=+0.503946770 container attach c9a6e1ebb1927380cca0d08635a5996af6eb8919b62327d25be763b3a7c3bb8b (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:32:01.432 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 6]}]': finished 2024-09-17T09:32:01.432 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 6]}]': finished 2024-09-17T09:32:01.432 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 0]}]': finished 2024-09-17T09:32:01.432 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-09-17T09:32:01.433 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 6]}]': finished 2024-09-17T09:32:01.433 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]': finished 2024-09-17T09:32:01.433 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 6]}]': finished 2024-09-17T09:32:01.433 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2, 4, 5]}]': finished 2024-09-17T09:32:01.433 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]': finished 2024-09-17T09:32:01.433 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 6]}]': finished 2024-09-17T09:32:01.433 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-mon[49967]: osdmap e110: 8 total, 7 up, 8 in 2024-09-17T09:32:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 6]}]': finished 2024-09-17T09:32:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 6]}]': finished 2024-09-17T09:32:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 0]}]': finished 2024-09-17T09:32:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-09-17T09:32:01.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 6]}]': finished 2024-09-17T09:32:01.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]': finished 2024-09-17T09:32:01.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 6]}]': finished 2024-09-17T09:32:01.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2, 4, 5]}]': finished 2024-09-17T09:32:01.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]': finished 2024-09-17T09:32:01.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 6]}]': finished 2024-09-17T09:32:01.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[58326]: osdmap e110: 8 total, 7 up, 8 in 2024-09-17T09:32:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 6]}]': finished 2024-09-17T09:32:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 6]}]': finished 2024-09-17T09:32:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [4, 0]}]': finished 2024-09-17T09:32:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-09-17T09:32:01.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 6]}]': finished 2024-09-17T09:32:01.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]': finished 2024-09-17T09:32:01.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 6]}]': finished 2024-09-17T09:32:01.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2, 4, 5]}]': finished 2024-09-17T09:32:01.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 6]}]': finished 2024-09-17T09:32:01.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 6]}]': finished 2024-09-17T09:32:01.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:01 smithi029 ceph-mon[60489]: osdmap e110: 8 total, 7 up, 8 in 2024-09-17T09:32:01.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-activate[58027]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T09:32:01.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 bash[58013]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T09:32:01.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-activate[58027]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T09:32:01.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 bash[58013]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T09:32:01.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-activate[58027]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T09:32:01.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 bash[58013]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T09:32:01.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-activate[58027]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T09:32:01.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 bash[58013]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T09:32:01.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-activate[58027]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-17T09:32:01.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 bash[58013]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-17T09:32:01.757 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-activate[58027]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T09:32:01.757 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 bash[58013]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T09:32:01.757 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-activate[58027]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-17T09:32:01.757 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 bash[58013]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-17T09:32:01.757 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:01 smithi103 podman[58013]: 2024-09-17 09:32:01.47874607 +0000 UTC m=+0.750154701 container died c9a6e1ebb1927380cca0d08635a5996af6eb8919b62327d25be763b3a7c3bb8b (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:32:02.346 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:02 smithi103 ceph-mon[49967]: pgmap v169: 161 pgs: 13 stale+active+clean, 148 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:32:02.346 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:02 smithi103 ceph-mon[49967]: osdmap e111: 8 total, 7 up, 8 in 2024-09-17T09:32:02.347 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:02 smithi103 podman[58156]: 2024-09-17 09:32:02.09474947 +0000 UTC m=+0.610734933 container remove c9a6e1ebb1927380cca0d08635a5996af6eb8919b62327d25be763b3a7c3bb8b (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-17T09:32:02.347 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:02 smithi103 podman[58216]: 2024-09-17 09:32:02.218081671 +0000 UTC m=+0.016121264 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:32:02.347 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:02 smithi103 podman[58216]: 2024-09-17 09:32:02.345832948 +0000 UTC m=+0.143872540 container create a5355c784b3cbd4793b7f4be63ac7313236a448c44c44c14c6939336919d5877 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b) 2024-09-17T09:32:02.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:02 smithi029 ceph-mon[58326]: pgmap v169: 161 pgs: 13 stale+active+clean, 148 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:32:02.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:02 smithi029 ceph-mon[58326]: osdmap e111: 8 total, 7 up, 8 in 2024-09-17T09:32:02.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:02 smithi029 ceph-mon[60489]: pgmap v169: 161 pgs: 13 stale+active+clean, 148 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:32:02.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:02 smithi029 ceph-mon[60489]: osdmap e111: 8 total, 7 up, 8 in 2024-09-17T09:32:03.006 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:02 smithi103 podman[58216]: 2024-09-17 09:32:02.607117139 +0000 UTC m=+0.405156741 container init a5355c784b3cbd4793b7f4be63ac7313236a448c44c44c14c6939336919d5877 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-17T09:32:03.006 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:02 smithi103 podman[58216]: 2024-09-17 09:32:02.610318184 +0000 UTC m=+0.408357777 container start a5355c784b3cbd4793b7f4be63ac7313236a448c44c44c14c6939336919d5877 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-17T09:32:03.006 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:02 smithi103 ceph-osd[58235]: -- 172.21.15.103:0/183453481 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559281d5b1e0 con 0x559281cf9c00 2024-09-17T09:32:03.006 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:02 smithi103 bash[58216]: a5355c784b3cbd4793b7f4be63ac7313236a448c44c44c14c6939336919d5877 2024-09-17T09:32:03.007 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:02 smithi103 systemd[1]: Started Ceph osd.6 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:32:03.437 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:32:03.145+0000 7f5b4e84c640 -1 Falling back to public interface 2024-09-17T09:32:03.438 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:03 smithi103 ceph-mon[49967]: osdmap e112: 8 total, 7 up, 8 in 2024-09-17T09:32:03.438 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:03 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:03.438 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:03 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:03.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:03 smithi029 ceph-mon[58326]: osdmap e112: 8 total, 7 up, 8 in 2024-09-17T09:32:03.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:03 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:03.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:03 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:03 smithi029 ceph-mon[60489]: osdmap e112: 8 total, 7 up, 8 in 2024-09-17T09:32:03.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:03 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:03.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:03 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:04.383 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:04 smithi103 ceph-mon[49967]: pgmap v172: 161 pgs: 2 activating+remapped, 3 remapped+peering, 2 peering, 27 active+undersized, 3 stale+active+clean, 13 active+undersized+degraded, 111 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 103/999 objects degraded (10.310%) 2024-09-17T09:32:04.384 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:04 smithi103 ceph-mon[49967]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-17T09:32:04.384 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:04 smithi103 ceph-mon[49967]: Health check failed: Degraded data redundancy: 103/999 objects degraded (10.310%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T09:32:04.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:04 smithi029 ceph-mon[60489]: pgmap v172: 161 pgs: 2 activating+remapped, 3 remapped+peering, 2 peering, 27 active+undersized, 3 stale+active+clean, 13 active+undersized+degraded, 111 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 103/999 objects degraded (10.310%) 2024-09-17T09:32:04.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:04 smithi029 ceph-mon[60489]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-17T09:32:04.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:04 smithi029 ceph-mon[60489]: Health check failed: Degraded data redundancy: 103/999 objects degraded (10.310%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T09:32:04.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:04 smithi029 ceph-mon[58326]: pgmap v172: 161 pgs: 2 activating+remapped, 3 remapped+peering, 2 peering, 27 active+undersized, 3 stale+active+clean, 13 active+undersized+degraded, 111 active+clean; 604 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 103/999 objects degraded (10.310%) 2024-09-17T09:32:04.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:04 smithi029 ceph-mon[58326]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-17T09:32:04.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:04 smithi029 ceph-mon[58326]: Health check failed: Degraded data redundancy: 103/999 objects degraded (10.310%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T09:32:06.112 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:32:05.858+0000 7f5b4e84c640 -1 osd.6 0 read_superblock omap replica is missing. 2024-09-17T09:32:06.112 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:32:05.888+0000 7f5b4e84c640 -1 osd.6 108 log_to_monitors true 2024-09-17T09:32:06.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:06 smithi103 ceph-mon[49967]: pgmap v173: 161 pgs: 3 active+clean+remapped, 2 activating+remapped, 3 remapped+peering, 2 peering, 32 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 122/999 objects degraded (12.212%) 2024-09-17T09:32:06.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:06 smithi103 ceph-mon[49967]: from='osd.6 [v2:172.21.15.103:6816/2282242951,v1:172.21.15.103:6817/2282242951]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:32:06.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:06 smithi103 ceph-mon[49967]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:32:06.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:06 smithi029 ceph-mon[58326]: pgmap v173: 161 pgs: 3 active+clean+remapped, 2 activating+remapped, 3 remapped+peering, 2 peering, 32 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 122/999 objects degraded (12.212%) 2024-09-17T09:32:06.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:06 smithi029 ceph-mon[58326]: from='osd.6 [v2:172.21.15.103:6816/2282242951,v1:172.21.15.103:6817/2282242951]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:32:06.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:06 smithi029 ceph-mon[58326]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:32:06.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:06 smithi029 ceph-mon[60489]: pgmap v173: 161 pgs: 3 active+clean+remapped, 2 activating+remapped, 3 remapped+peering, 2 peering, 32 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 122/999 objects degraded (12.212%) 2024-09-17T09:32:06.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:06 smithi029 ceph-mon[60489]: from='osd.6 [v2:172.21.15.103:6816/2282242951,v1:172.21.15.103:6817/2282242951]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:32:06.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:06 smithi029 ceph-mon[60489]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T09:32:07.005 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:32:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:32:06.580+0000 7f5b425ee640 -1 osd.6 108 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:32:07.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:07 smithi103 ceph-mon[49967]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T09:32:07.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:07 smithi103 ceph-mon[49967]: osdmap e113: 8 total, 7 up, 8 in 2024-09-17T09:32:07.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:07 smithi103 ceph-mon[49967]: from='osd.6 [v2:172.21.15.103:6816/2282242951,v1:172.21.15.103:6817/2282242951]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:32:07.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:07 smithi103 ceph-mon[49967]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:32:07.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:07 smithi103 ceph-mon[49967]: from='osd.6 ' entity='osd.6' 2024-09-17T09:32:07.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:07 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:07.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:07 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:07.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[60489]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T09:32:07.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[60489]: osdmap e113: 8 total, 7 up, 8 in 2024-09-17T09:32:07.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[60489]: from='osd.6 [v2:172.21.15.103:6816/2282242951,v1:172.21.15.103:6817/2282242951]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:32:07.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[60489]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:32:07.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[60489]: from='osd.6 ' entity='osd.6' 2024-09-17T09:32:07.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:07.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:07.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[58326]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T09:32:07.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[58326]: osdmap e113: 8 total, 7 up, 8 in 2024-09-17T09:32:07.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[58326]: from='osd.6 [v2:172.21.15.103:6816/2282242951,v1:172.21.15.103:6817/2282242951]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:32:07.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[58326]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:32:07.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[58326]: from='osd.6 ' entity='osd.6' 2024-09-17T09:32:07.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:07.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:07 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:08.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:08 smithi103 ceph-mon[49967]: pgmap v175: 161 pgs: 5 active+clean+remapped, 2 activating+remapped, 3 remapped+peering, 32 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 122/999 objects degraded (12.212%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:08.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:08.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:08.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:08 smithi103 ceph-mon[49967]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:32:08.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:08 smithi103 ceph-mon[49967]: osd.6 [v2:172.21.15.103:6816/2282242951,v1:172.21.15.103:6817/2282242951] boot 2024-09-17T09:32:08.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:08 smithi103 ceph-mon[49967]: osdmap e114: 8 total, 8 up, 8 in 2024-09-17T09:32:08.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:08 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:32:08.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:08 smithi103 ceph-mon[49967]: osdmap e115: 8 total, 8 up, 8 in 2024-09-17T09:32:08.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[60489]: pgmap v175: 161 pgs: 5 active+clean+remapped, 2 activating+remapped, 3 remapped+peering, 32 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 122/999 objects degraded (12.212%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:08.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:08.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:08.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[60489]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:32:08.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[60489]: osd.6 [v2:172.21.15.103:6816/2282242951,v1:172.21.15.103:6817/2282242951] boot 2024-09-17T09:32:08.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[60489]: osdmap e114: 8 total, 8 up, 8 in 2024-09-17T09:32:08.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:32:08.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[60489]: osdmap e115: 8 total, 8 up, 8 in 2024-09-17T09:32:08.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[58326]: pgmap v175: 161 pgs: 5 active+clean+remapped, 2 activating+remapped, 3 remapped+peering, 32 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 122/999 objects degraded (12.212%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:08.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:08.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:08.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[58326]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:32:08.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[58326]: osd.6 [v2:172.21.15.103:6816/2282242951,v1:172.21.15.103:6817/2282242951] boot 2024-09-17T09:32:08.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[58326]: osdmap e114: 8 total, 8 up, 8 in 2024-09-17T09:32:08.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T09:32:08.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:08 smithi029 ceph-mon[58326]: osdmap e115: 8 total, 8 up, 8 in 2024-09-17T09:32:10.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:09 smithi029 ceph-mon[60489]: pgmap v178: 161 pgs: 10 active+clean+remapped, 32 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 122/999 objects degraded (12.212%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:10.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:09 smithi029 ceph-mon[60489]: osdmap e116: 8 total, 8 up, 8 in 2024-09-17T09:32:10.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:09 smithi029 ceph-mon[60489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 2 pgs peering) 2024-09-17T09:32:10.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:09 smithi029 ceph-mon[58326]: pgmap v178: 161 pgs: 10 active+clean+remapped, 32 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 122/999 objects degraded (12.212%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:10.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:09 smithi029 ceph-mon[58326]: osdmap e116: 8 total, 8 up, 8 in 2024-09-17T09:32:10.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:09 smithi029 ceph-mon[58326]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 2 pgs peering) 2024-09-17T09:32:10.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:09 smithi103 ceph-mon[49967]: pgmap v178: 161 pgs: 10 active+clean+remapped, 32 active+undersized, 17 active+undersized+degraded, 102 active+clean; 604 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 122/999 objects degraded (12.212%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:10.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:09 smithi103 ceph-mon[49967]: osdmap e116: 8 total, 8 up, 8 in 2024-09-17T09:32:10.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:09 smithi103 ceph-mon[49967]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 2 pgs peering) 2024-09-17T09:32:11.048 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:32:10 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:32:10.759+0000 7fe73b7e7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-17T09:32:12.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: pgmap v180: 161 pgs: 9 active+clean+remapped, 27 active+undersized, 15 active+undersized+degraded, 110 active+clean; 604 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:12.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:12.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:12.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:12.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:32:12.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:32:12.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: pgmap v181: 161 pgs: 9 active+clean+remapped, 21 active+undersized, 12 active+undersized+degraded, 119 active+clean; 604 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 83/999 objects degraded (8.308%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:12.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: pgmap v182: 161 pgs: 9 active+clean+remapped, 21 active+undersized, 12 active+undersized+degraded, 119 active+clean; 604 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 83/999 objects degraded (8.308%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:12.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:12.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:32:12.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:12.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: Health check update: Degraded data redundancy: 83/999 objects degraded (8.308%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T09:32:12.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:12.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:12.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T09:32:12.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T09:32:12.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:11 smithi103 ceph-mon[49967]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-17T09:32:12.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: pgmap v180: 161 pgs: 9 active+clean+remapped, 27 active+undersized, 15 active+undersized+degraded, 110 active+clean; 604 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:12.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:12.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:12.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:12.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:32:12.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:32:12.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: pgmap v181: 161 pgs: 9 active+clean+remapped, 21 active+undersized, 12 active+undersized+degraded, 119 active+clean; 604 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 83/999 objects degraded (8.308%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:12.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: pgmap v182: 161 pgs: 9 active+clean+remapped, 21 active+undersized, 12 active+undersized+degraded, 119 active+clean; 604 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 83/999 objects degraded (8.308%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:12.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:12.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:32:12.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:12.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: Health check update: Degraded data redundancy: 83/999 objects degraded (8.308%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T09:32:12.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:12.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:12.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T09:32:12.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T09:32:12.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[58326]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-17T09:32:12.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: pgmap v180: 161 pgs: 9 active+clean+remapped, 27 active+undersized, 15 active+undersized+degraded, 110 active+clean; 604 KiB data, 275 MiB used, 715 GiB / 715 GiB avail; 105/999 objects degraded (10.511%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:12.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:12.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:12.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:12.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:32:12.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:32:12.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: pgmap v181: 161 pgs: 9 active+clean+remapped, 21 active+undersized, 12 active+undersized+degraded, 119 active+clean; 604 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 83/999 objects degraded (8.308%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:12.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: pgmap v182: 161 pgs: 9 active+clean+remapped, 21 active+undersized, 12 active+undersized+degraded, 119 active+clean; 604 KiB data, 276 MiB used, 715 GiB / 715 GiB avail; 83/999 objects degraded (8.308%); 1/999 objects misplaced (0.100%) 2024-09-17T09:32:12.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:12.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:32:12.052 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:12.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: Health check update: Degraded data redundancy: 83/999 objects degraded (8.308%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T09:32:12.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:12.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:12.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T09:32:12.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T09:32:12.053 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:11 smithi029 ceph-mon[60489]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-17T09:32:13.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:12 smithi103 ceph-mon[49967]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 83/999 objects degraded (8.308%), 12 pgs degraded) 2024-09-17T09:32:13.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:12 smithi029 ceph-mon[58326]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 83/999 objects degraded (8.308%), 12 pgs degraded) 2024-09-17T09:32:13.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:12 smithi029 ceph-mon[60489]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 83/999 objects degraded (8.308%), 12 pgs degraded) 2024-09-17T09:32:14.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:13 smithi103 ceph-mon[49967]: pgmap v183: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:14.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:13 smithi029 ceph-mon[58326]: pgmap v183: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:14.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:13 smithi029 ceph-mon[60489]: pgmap v183: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:15.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:15 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:15.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:15 smithi103 ceph-mon[49967]: pgmap v184: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:15.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:15 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:15.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:15 smithi029 ceph-mon[58326]: pgmap v184: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:15.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:15 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:15.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:15 smithi029 ceph-mon[60489]: pgmap v184: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:18.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:17 smithi103 ceph-mon[49967]: pgmap v185: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:18.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:17 smithi029 ceph-mon[58326]: pgmap v185: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:18.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:17 smithi029 ceph-mon[60489]: pgmap v185: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:20.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:19 smithi103 ceph-mon[49967]: pgmap v186: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:20.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:19 smithi029 ceph-mon[58326]: pgmap v186: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:20.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:19 smithi029 ceph-mon[60489]: pgmap v186: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:22.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:21 smithi103 ceph-mon[49967]: pgmap v187: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:22.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:21 smithi029 ceph-mon[58326]: pgmap v187: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:22.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:21 smithi029 ceph-mon[60489]: pgmap v187: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:23.346 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:32:24.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:23 smithi103 ceph-mon[49967]: pgmap v188: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:24.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:23 smithi029 ceph-mon[58326]: pgmap v188: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:24.010 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:23 smithi029 ceph-mon[60489]: pgmap v188: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-17T09:32:24.011 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:32:24.011 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (6m) 100s ago 10m 19.1M - 0.20.0 0881eb8f169f 7bd158a6d65c 2024-09-17T09:32:24.011 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (5m) 17s ago 10m 36.9M - 6.7.4 557c83e11646 e361816f7c14 2024-09-17T09:32:24.011 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 running (102s) 100s ago 9m - - 24ecd6d5f14c 8d25644a0887 2024-09-17T09:32:24.011 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (4m) 17s ago 14m 438M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:32:24.011 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (5m) 100s ago 15m 485M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:32:24.011 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (4m) 100s ago 16m 49.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:32:24.011 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (3m) 17s ago 14m 41.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 da050285f124 2024-09-17T09:32:24.011 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (4m) 100s ago 14m 45.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 0d3d1a65e2c9 2024-09-17T09:32:24.011 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (6m) 100s ago 11m 15.0M - 0.18.1 e5a616e4b9cf 34bfa263169f 2024-09-17T09:32:24.011 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (5m) 17s ago 11m 8866k - 0.18.1 e5a616e4b9cf 1adf1b5baad1 2024-09-17T09:32:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (3m) 100s ago 14m 52.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 4c456435000b 2024-09-17T09:32:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (2m) 100s ago 13m 44.3M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2ff250ddb1d9 2024-09-17T09:32:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (2m) 100s ago 13m 49.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7899f3c3c2f4 2024-09-17T09:32:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (106s) 100s ago 13m 12.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 94b84aca13ce 2024-09-17T09:32:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (87s) 17s ago 12m 44.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2306402255ea 2024-09-17T09:32:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (54s) 17s ago 12m 43.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0312d071d90b 2024-09-17T09:32:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (21s) 17s ago 12m 11.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 a5355c784b3c 2024-09-17T09:32:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (12m) 17s ago 12m 58.7M 4096M 16.2.0 24ecd6d5f14c 0b57b23d30fa 2024-09-17T09:32:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 error 17s ago 11m - - 2024-09-17T09:32:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (9m) 100s ago 9m 41.6M - 16.2.0 24ecd6d5f14c 715c2ccb6075 2024-09-17T09:32:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (9m) 17s ago 9m 41.6M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:32:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (9m) 100s ago 9m 39.3M - 16.2.0 24ecd6d5f14c 6b9ad943cb9e 2024-09-17T09:32:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (9m) 17s ago 9m 39.2M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:32:24.416 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:32:24.416 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:32:24.416 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:32:24.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:32:24.417 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:32:24.417 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:32:24.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:32:24.417 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:32:24.417 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 1, 2024-09-17T09:32:24.417 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 7 2024-09-17T09:32:24.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:32:24.417 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:32:24.417 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 4 2024-09-17T09:32:24.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:32:24.417 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:32:24.417 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 5, 2024-09-17T09:32:24.417 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 12 2024-09-17T09:32:24.418 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:32:24.418 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:32:24.756 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:32:24.756 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:32:24.756 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:32:24.756 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:32:24.756 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [ 2024-09-17T09:32:24.756 INFO:teuthology.orchestra.run.smithi029.stdout: "mon", 2024-09-17T09:32:24.756 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr" 2024-09-17T09:32:24.756 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:32:24.757 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "12/23 daemons upgraded", 2024-09-17T09:32:24.757 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T09:32:24.757 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:32:24.757 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:32:25.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:24 smithi103 ceph-mon[49967]: from='client.54745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:25.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:24 smithi103 ceph-mon[49967]: from='client.44871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:25.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:24 smithi103 ceph-mon[49967]: from='client.44877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:25.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:24 smithi103 ceph-mon[49967]: from='client.44883 172.21.15.29:0/2382304594' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:25.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:24 smithi029 ceph-mon[58326]: from='client.54745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:25.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:24 smithi029 ceph-mon[58326]: from='client.44871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:25.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:24 smithi029 ceph-mon[58326]: from='client.44877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:25.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:24 smithi029 ceph-mon[58326]: from='client.44883 172.21.15.29:0/2382304594' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:25.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:24 smithi029 ceph-mon[60489]: from='client.54745 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:25.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:24 smithi029 ceph-mon[60489]: from='client.44871 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:25.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:24 smithi029 ceph-mon[60489]: from='client.44877 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:25.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:24 smithi029 ceph-mon[60489]: from='client.44883 172.21.15.29:0/2382304594' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:25.156 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T09:32:25.156 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T09:32:25.156 INFO:teuthology.orchestra.run.smithi029.stdout: daemon prometheus.a on smithi103 is in error state 2024-09-17T09:32:26.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:25 smithi103 ceph-mon[49967]: pgmap v189: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:32:26.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:25 smithi103 ceph-mon[49967]: from='client.54769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:26.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:25 smithi103 ceph-mon[49967]: from='client.44895 172.21.15.29:0/240852847' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:32:26.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:25 smithi029 ceph-mon[58326]: pgmap v189: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:32:26.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:25 smithi029 ceph-mon[58326]: from='client.54769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:26.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:25 smithi029 ceph-mon[58326]: from='client.44895 172.21.15.29:0/240852847' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:32:26.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:25 smithi029 ceph-mon[60489]: pgmap v189: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:32:26.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:25 smithi029 ceph-mon[60489]: from='client.54769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:26.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:25 smithi029 ceph-mon[60489]: from='client.44895 172.21.15.29:0/240852847' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:32:27.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T09:32:27.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:26 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T09:32:27.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:26 smithi103 ceph-mon[49967]: Upgrade: osd.7 is safe to restart 2024-09-17T09:32:27.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T09:32:27.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:26 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T09:32:27.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:26 smithi029 ceph-mon[60489]: Upgrade: osd.7 is safe to restart 2024-09-17T09:32:27.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T09:32:27.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:26 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T09:32:27.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:26 smithi029 ceph-mon[58326]: Upgrade: osd.7 is safe to restart 2024-09-17T09:32:28.441 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:28 smithi103 ceph-mon[49967]: pgmap v190: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:32:28.441 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:28 smithi103 ceph-mon[49967]: Upgrade: Updating osd.7 2024-09-17T09:32:28.441 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:28.441 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T09:32:28.441 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:28.441 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:28 smithi103 ceph-mon[49967]: Deploying daemon osd.7 on smithi103 2024-09-17T09:32:28.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:28 smithi029 ceph-mon[58326]: pgmap v190: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:32:28.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:28 smithi029 ceph-mon[58326]: Upgrade: Updating osd.7 2024-09-17T09:32:28.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:28.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T09:32:28.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:28.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:28 smithi029 ceph-mon[58326]: Deploying daemon osd.7 on smithi103 2024-09-17T09:32:28.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:28 smithi029 ceph-mon[60489]: pgmap v190: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:32:28.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:28 smithi029 ceph-mon[60489]: Upgrade: Updating osd.7 2024-09-17T09:32:28.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:28.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T09:32:28.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:28.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:28 smithi029 ceph-mon[60489]: Deploying daemon osd.7 on smithi103 2024-09-17T09:32:30.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:30 smithi103 ceph-mon[49967]: pgmap v191: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-17T09:32:30.507 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:30 smithi103 systemd[1]: Stopping Ceph osd.7 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:32:30.507 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.7[33757]: debug 2024-09-17T09:32:30.261+0000 7f03ba14d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-17T09:32:30.507 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.7[33757]: debug 2024-09-17T09:32:30.261+0000 7f03ba14d700 -1 osd.7 116 *** Got signal Terminated *** 2024-09-17T09:32:30.507 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:30 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.7[33757]: debug 2024-09-17T09:32:30.261+0000 7f03ba14d700 -1 osd.7 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:32:30.507 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:30 smithi103 podman[60105]: 2024-09-17 09:32:30.293512559 +0000 UTC m=+0.155561606 container died 0b57b23d30fa247d5bab0f8ea77ac46e174762b0db9d363fe71db177d9046fc5 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.7, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, ceph=True, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) 2024-09-17T09:32:30.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:30 smithi029 ceph-mon[58326]: pgmap v191: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-17T09:32:30.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:30 smithi029 ceph-mon[60489]: pgmap v191: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-17T09:32:31.005 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:30 smithi103 podman[60105]: 2024-09-17 09:32:30.762787257 +0000 UTC m=+0.624836303 container cleanup 0b57b23d30fa247d5bab0f8ea77ac46e174762b0db9d363fe71db177d9046fc5 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-16.2.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, ceph=True, maintainer=Dimitri Savineau , org.label-schema.name=CentOS Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-09-17T09:32:31.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:30 smithi103 bash[60105]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.7 2024-09-17T09:32:31.486 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:31 smithi103 podman[60117]: 2024-09-17 09:32:31.031223017 +0000 UTC m=+0.733050918 container remove 0b57b23d30fa247d5bab0f8ea77ac46e174762b0db9d363fe71db177d9046fc5 (image=quay.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e00951949000, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd.7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-16.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=20cb635043d86d971f028e2bf2824446440b5557) 2024-09-17T09:32:31.486 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.486 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 failed (root=default,host=smithi103) (connection refused reported by osd.2) 2024-09-17T09:32:31.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.5 2024-09-17T09:32:31.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.5 2024-09-17T09:32:31.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.6 2024-09-17T09:32:31.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.6 2024-09-17T09:32:31.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.1 2024-09-17T09:32:31.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.1 2024-09-17T09:32:31.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.6 2024-09-17T09:32:31.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.5 2024-09-17T09:32:31.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.6 2024-09-17T09:32:31.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.488 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.5 2024-09-17T09:32:31.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.1 2024-09-17T09:32:31.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.1 2024-09-17T09:32:31.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:31 smithi103 ceph-mon[49967]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 failed (root=default,host=smithi103) (connection refused reported by osd.2) 2024-09-17T09:32:31.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.5 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.5 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.6 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.6 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.1 2024-09-17T09:32:31.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.1 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.6 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.5 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.6 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.5 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.1 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.1 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[58326]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 failed (root=default,host=smithi103) (connection refused reported by osd.2) 2024-09-17T09:32:31.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.5 2024-09-17T09:32:31.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.5 2024-09-17T09:32:31.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.6 2024-09-17T09:32:31.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.6 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.1 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.1 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.6 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.5 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.6 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.5 2024-09-17T09:32:31.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.1 2024-09-17T09:32:31.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.1 2024-09-17T09:32:31.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.2 2024-09-17T09:32:31.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.4 2024-09-17T09:32:31.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.0 2024-09-17T09:32:31.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:31 smithi029 ceph-mon[60489]: osd.7 reported immediately failed by osd.3 2024-09-17T09:32:31.739 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:31 smithi103 podman[60221]: 2024-09-17 09:32:31.388103059 +0000 UTC m=+0.015998865 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:32:31.740 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:31 smithi103 podman[60221]: 2024-09-17 09:32:31.503690505 +0000 UTC m=+0.131586353 container create 39062772bbfd0781d775fdfbd05dc3d12280396c7fdb7b067f5c755378e54765 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-deactivate, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-17T09:32:31.740 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:31 smithi103 podman[60221]: 2024-09-17 09:32:31.735163513 +0000 UTC m=+0.363059316 container init 39062772bbfd0781d775fdfbd05dc3d12280396c7fdb7b067f5c755378e54765 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:32:31.740 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:31 smithi103 podman[60221]: 2024-09-17 09:32:31.738973906 +0000 UTC m=+0.366869710 container start 39062772bbfd0781d775fdfbd05dc3d12280396c7fdb7b067f5c755378e54765 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T09:32:32.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:31 smithi103 podman[60221]: 2024-09-17 09:32:31.852133731 +0000 UTC m=+0.480029537 container attach 39062772bbfd0781d775fdfbd05dc3d12280396c7fdb7b067f5c755378e54765 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b) 2024-09-17T09:32:32.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:31 smithi103 podman[60221]: 2024-09-17 09:32:31.928536403 +0000 UTC m=+0.556432207 container died 39062772bbfd0781d775fdfbd05dc3d12280396c7fdb7b067f5c755378e54765 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-17T09:32:32.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:32 smithi103 ceph-mon[49967]: pgmap v192: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-17T09:32:32.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:32 smithi103 ceph-mon[49967]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:32:32.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:32 smithi103 ceph-mon[49967]: Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2024-09-17T09:32:32.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:32 smithi103 ceph-mon[49967]: osdmap e117: 8 total, 7 up, 8 in 2024-09-17T09:32:32.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:32 smithi029 ceph-mon[58326]: pgmap v192: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-17T09:32:32.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:32 smithi029 ceph-mon[58326]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:32:32.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:32 smithi029 ceph-mon[58326]: Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2024-09-17T09:32:32.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:32 smithi029 ceph-mon[58326]: osdmap e117: 8 total, 7 up, 8 in 2024-09-17T09:32:32.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:32 smithi029 ceph-mon[60489]: pgmap v192: 161 pgs: 161 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-17T09:32:32.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:32 smithi029 ceph-mon[60489]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T09:32:32.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:32 smithi029 ceph-mon[60489]: Health check failed: all OSDs are running quincy or later but require_osd_release < quincy (OSD_UPGRADE_FINISHED) 2024-09-17T09:32:32.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:32 smithi029 ceph-mon[60489]: osdmap e117: 8 total, 7 up, 8 in 2024-09-17T09:32:32.846 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:32 smithi103 podman[60244]: 2024-09-17 09:32:32.62396299 +0000 UTC m=+0.685766726 container remove 39062772bbfd0781d775fdfbd05dc3d12280396c7fdb7b067f5c755378e54765 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-17T09:32:32.846 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:32 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.7.service: Deactivated successfully. 2024-09-17T09:32:32.846 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:32 smithi103 systemd[1]: Stopped Ceph osd.7 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:32:32.846 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:32 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.7.service: Consumed 1.104s CPU time. 2024-09-17T09:32:33.167 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:32 smithi103 systemd[1]: Starting Ceph osd.7 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:32:33.167 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 podman[60342]: 2024-09-17 09:32:32.958060814 +0000 UTC m=+0.016524782 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:32:33.167 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 podman[60342]: 2024-09-17 09:32:33.058582372 +0000 UTC m=+0.117046335 container create 16a27b3d10fd59640b54d15a8cf3b9883d6c204376b7dc91629583a97045200a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-activate, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b) 2024-09-17T09:32:33.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:33 smithi103 ceph-mon[49967]: osdmap e118: 8 total, 7 up, 8 in 2024-09-17T09:32:33.506 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 podman[60342]: 2024-09-17 09:32:33.29466226 +0000 UTC m=+0.353126241 container init 16a27b3d10fd59640b54d15a8cf3b9883d6c204376b7dc91629583a97045200a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-17T09:32:33.506 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 podman[60342]: 2024-09-17 09:32:33.298177287 +0000 UTC m=+0.356641253 container start 16a27b3d10fd59640b54d15a8cf3b9883d6c204376b7dc91629583a97045200a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-17T09:32:33.506 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 podman[60342]: 2024-09-17 09:32:33.377889303 +0000 UTC m=+0.436353282 container attach 16a27b3d10fd59640b54d15a8cf3b9883d6c204376b7dc91629583a97045200a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-activate, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-17T09:32:33.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:33 smithi029 ceph-mon[58326]: osdmap e118: 8 total, 7 up, 8 in 2024-09-17T09:32:33.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:33 smithi029 ceph-mon[60489]: osdmap e118: 8 total, 7 up, 8 in 2024-09-17T09:32:33.950 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-activate[60357]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T09:32:33.950 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 bash[60342]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T09:32:33.950 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-activate[60357]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T09:32:33.950 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 bash[60342]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T09:32:33.950 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-activate[60357]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T09:32:33.950 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 bash[60342]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T09:32:33.950 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-activate[60357]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T09:32:33.950 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 bash[60342]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T09:32:33.950 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-activate[60357]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-17T09:32:33.951 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 bash[60342]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-17T09:32:33.951 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-activate[60357]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T09:32:33.951 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 bash[60342]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T09:32:33.951 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-activate[60357]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-17T09:32:33.951 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 bash[60342]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-17T09:32:33.951 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:33 smithi103 podman[60342]: 2024-09-17 09:32:33.639813872 +0000 UTC m=+0.698277838 container died 16a27b3d10fd59640b54d15a8cf3b9883d6c204376b7dc91629583a97045200a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-17T09:32:34.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:34 smithi103 ceph-mon[49967]: pgmap v195: 161 pgs: 31 peering, 16 stale+active+clean, 114 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-17T09:32:34.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:34 smithi103 ceph-mon[49967]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-17T09:32:34.506 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:34 smithi103 podman[60485]: 2024-09-17 09:32:34.349468116 +0000 UTC m=+0.703324214 container remove 16a27b3d10fd59640b54d15a8cf3b9883d6c204376b7dc91629583a97045200a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-activate, GIT_CLEAN=True, RELEASE=reef-19e2a2b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-17T09:32:34.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:34 smithi029 ceph-mon[58326]: pgmap v195: 161 pgs: 31 peering, 16 stale+active+clean, 114 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-17T09:32:34.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:34 smithi029 ceph-mon[58326]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-17T09:32:34.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:34 smithi029 ceph-mon[60489]: pgmap v195: 161 pgs: 31 peering, 16 stale+active+clean, 114 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 129 op/s 2024-09-17T09:32:34.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:34 smithi029 ceph-mon[60489]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-17T09:32:34.862 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:34 smithi103 podman[60544]: 2024-09-17 09:32:34.550617937 +0000 UTC m=+0.106191437 container create 02f285335edfd31748e735f331423532a5bbadd35b3664fa7c05abd9dade99f6 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:32:34.862 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:34 smithi103 podman[60544]: 2024-09-17 09:32:34.460240652 +0000 UTC m=+0.015814119 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:32:34.862 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:34 smithi103 podman[60544]: 2024-09-17 09:32:34.761125092 +0000 UTC m=+0.316698554 container init 02f285335edfd31748e735f331423532a5bbadd35b3664fa7c05abd9dade99f6 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:32:34.862 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:34 smithi103 podman[60544]: 2024-09-17 09:32:34.76428528 +0000 UTC m=+0.319858737 container start 02f285335edfd31748e735f331423532a5bbadd35b3664fa7c05abd9dade99f6 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:32:34.863 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:34 smithi103 ceph-osd[60562]: -- 172.21.15.103:0/2423729845 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bbd0ad51e0 con 0x55bbd0a73c00 2024-09-17T09:32:34.863 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:34 smithi103 bash[60544]: 02f285335edfd31748e735f331423532a5bbadd35b3664fa7c05abd9dade99f6 2024-09-17T09:32:35.255 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:34 smithi103 systemd[1]: Started Ceph osd.7 for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:32:35.589 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:35 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:32:35.310+0000 7f86e4b88640 -1 Falling back to public interface 2024-09-17T09:32:36.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:35 smithi103 ceph-mon[49967]: pgmap v196: 161 pgs: 6 active+undersized, 31 peering, 13 stale+active+clean, 4 active+undersized+degraded, 107 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 0 B/s wr, 171 op/s; 25/999 objects degraded (2.503%) 2024-09-17T09:32:36.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:35 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:36.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:35 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:36.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:35 smithi103 ceph-mon[49967]: Health check failed: Degraded data redundancy: 25/999 objects degraded (2.503%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T09:32:36.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:35 smithi029 ceph-mon[58326]: pgmap v196: 161 pgs: 6 active+undersized, 31 peering, 13 stale+active+clean, 4 active+undersized+degraded, 107 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 0 B/s wr, 171 op/s; 25/999 objects degraded (2.503%) 2024-09-17T09:32:36.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:35 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:36.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:35 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:36.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:35 smithi029 ceph-mon[58326]: Health check failed: Degraded data redundancy: 25/999 objects degraded (2.503%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T09:32:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:35 smithi029 ceph-mon[60489]: pgmap v196: 161 pgs: 6 active+undersized, 31 peering, 13 stale+active+clean, 4 active+undersized+degraded, 107 active+clean; 604 KiB data, 280 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 0 B/s wr, 171 op/s; 25/999 objects degraded (2.503%) 2024-09-17T09:32:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:35 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:35 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:36.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:35 smithi029 ceph-mon[60489]: Health check failed: Degraded data redundancy: 25/999 objects degraded (2.503%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T09:32:38.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:37 smithi103 ceph-mon[49967]: pgmap v197: 161 pgs: 30 active+undersized, 18 peering, 21 active+undersized+degraded, 92 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 0 B/s wr, 281 op/s; 103/999 objects degraded (10.310%) 2024-09-17T09:32:38.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:32:38.031+0000 7f86e4b88640 -1 osd.7 0 read_superblock omap replica is missing. 2024-09-17T09:32:38.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:32:38.063+0000 7f86e4b88640 -1 osd.7 116 log_to_monitors true 2024-09-17T09:32:38.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:37 smithi029 ceph-mon[58326]: pgmap v197: 161 pgs: 30 active+undersized, 18 peering, 21 active+undersized+degraded, 92 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 0 B/s wr, 281 op/s; 103/999 objects degraded (10.310%) 2024-09-17T09:32:38.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:37 smithi029 ceph-mon[60489]: pgmap v197: 161 pgs: 30 active+undersized, 18 peering, 21 active+undersized+degraded, 92 active+clean; 604 KiB data, 284 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 0 B/s wr, 281 op/s; 103/999 objects degraded (10.310%) 2024-09-17T09:32:39.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:38 smithi103 ceph-mon[49967]: from='osd.7 [v2:172.21.15.103:6824/4008210445,v1:172.21.15.103:6825/4008210445]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:32:39.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:38 smithi103 ceph-mon[49967]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:32:39.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:32:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:32:38.929+0000 7f86dc932640 -1 osd.7 116 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T09:32:39.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:38 smithi029 ceph-mon[58326]: from='osd.7 [v2:172.21.15.103:6824/4008210445,v1:172.21.15.103:6825/4008210445]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:32:39.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:38 smithi029 ceph-mon[58326]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:32:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:38 smithi029 ceph-mon[60489]: from='osd.7 [v2:172.21.15.103:6824/4008210445,v1:172.21.15.103:6825/4008210445]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:32:39.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:38 smithi029 ceph-mon[60489]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T09:32:40.065 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:39 smithi103 ceph-mon[49967]: pgmap v198: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 288 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 0 B/s wr, 300 op/s; 131/999 objects degraded (13.113%) 2024-09-17T09:32:40.065 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:39 smithi103 ceph-mon[49967]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-17T09:32:40.065 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:39 smithi103 ceph-mon[49967]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T09:32:40.065 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:39 smithi103 ceph-mon[49967]: osdmap e119: 8 total, 7 up, 8 in 2024-09-17T09:32:40.065 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:39 smithi103 ceph-mon[49967]: from='osd.7 [v2:172.21.15.103:6824/4008210445,v1:172.21.15.103:6825/4008210445]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:32:40.065 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:39 smithi103 ceph-mon[49967]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:32:40.065 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:39 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:40.065 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:39 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:40.066 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:39 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:40.066 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:39 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:40.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[60489]: pgmap v198: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 288 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 0 B/s wr, 300 op/s; 131/999 objects degraded (13.113%) 2024-09-17T09:32:40.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[60489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-17T09:32:40.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[60489]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T09:32:40.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[60489]: osdmap e119: 8 total, 7 up, 8 in 2024-09-17T09:32:40.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[60489]: from='osd.7 [v2:172.21.15.103:6824/4008210445,v1:172.21.15.103:6825/4008210445]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:32:40.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[60489]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:32:40.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:40.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:40.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:40.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:40.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[58326]: pgmap v198: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 288 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 0 B/s wr, 300 op/s; 131/999 objects degraded (13.113%) 2024-09-17T09:32:40.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[58326]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-17T09:32:40.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[58326]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T09:32:40.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[58326]: osdmap e119: 8 total, 7 up, 8 in 2024-09-17T09:32:40.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[58326]: from='osd.7 [v2:172.21.15.103:6824/4008210445,v1:172.21.15.103:6825/4008210445]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:32:40.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[58326]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-09-17T09:32:40.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:40.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:40.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:40.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:39 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:41.145 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:40 smithi103 ceph-mon[49967]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:32:41.145 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:40 smithi103 ceph-mon[49967]: osd.7 [v2:172.21.15.103:6824/4008210445,v1:172.21.15.103:6825/4008210445] boot 2024-09-17T09:32:41.146 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:40 smithi103 ceph-mon[49967]: osdmap e120: 8 total, 8 up, 8 in 2024-09-17T09:32:41.146 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:40 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:32:41.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:40 smithi029 ceph-mon[58326]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:32:41.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:40 smithi029 ceph-mon[58326]: osd.7 [v2:172.21.15.103:6824/4008210445,v1:172.21.15.103:6825/4008210445] boot 2024-09-17T09:32:41.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:40 smithi029 ceph-mon[58326]: osdmap e120: 8 total, 8 up, 8 in 2024-09-17T09:32:41.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:40 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:32:41.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:40 smithi029 ceph-mon[60489]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T09:32:41.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:40 smithi029 ceph-mon[60489]: osd.7 [v2:172.21.15.103:6824/4008210445,v1:172.21.15.103:6825/4008210445] boot 2024-09-17T09:32:41.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:40 smithi029 ceph-mon[60489]: osdmap e120: 8 total, 8 up, 8 in 2024-09-17T09:32:41.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:40 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T09:32:42.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:41 smithi103 ceph-mon[49967]: pgmap v201: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 0 B/s wr, 211 op/s; 131/999 objects degraded (13.113%) 2024-09-17T09:32:42.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:41 smithi103 ceph-mon[49967]: osdmap e121: 8 total, 8 up, 8 in 2024-09-17T09:32:42.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:41 smithi029 ceph-mon[58326]: pgmap v201: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 0 B/s wr, 211 op/s; 131/999 objects degraded (13.113%) 2024-09-17T09:32:42.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:41 smithi029 ceph-mon[58326]: osdmap e121: 8 total, 8 up, 8 in 2024-09-17T09:32:42.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:41 smithi029 ceph-mon[60489]: pgmap v201: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 0 B/s wr, 211 op/s; 131/999 objects degraded (13.113%) 2024-09-17T09:32:42.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:41 smithi029 ceph-mon[60489]: osdmap e121: 8 total, 8 up, 8 in 2024-09-17T09:32:43.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:42 smithi103 ceph-mon[49967]: Health check update: Degraded data redundancy: 131/999 objects degraded (13.113%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T09:32:43.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:42 smithi029 ceph-mon[58326]: Health check update: Degraded data redundancy: 131/999 objects degraded (13.113%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T09:32:43.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:42 smithi029 ceph-mon[60489]: Health check update: Degraded data redundancy: 131/999 objects degraded (13.113%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T09:32:44.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: pgmap v203: 161 pgs: 18 active+undersized, 12 active+undersized+degraded, 131 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s; 72/999 objects degraded (7.207%); 0 B/s, 3 objects/s recovering 2024-09-17T09:32:44.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:44.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:44.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:44.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: pgmap v204: 161 pgs: 18 active+undersized, 12 active+undersized+degraded, 131 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s; 72/999 objects degraded (7.207%); 0 B/s, 4 objects/s recovering 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: pgmap v205: 161 pgs: 18 active+undersized, 12 active+undersized+degraded, 131 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 44 op/s; 72/999 objects degraded (7.207%); 0 B/s, 4 objects/s recovering 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: Upgrade: Setting container_image for all osd 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-17T09:32:44.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-17T09:32:44.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-17T09:32:44.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-17T09:32:44.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-17T09:32:44.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-17T09:32:44.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-17T09:32:44.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-17T09:32:44.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-17T09:32:44.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-17T09:32:44.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-17T09:32:44.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-17T09:32:44.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-17T09:32:44.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-17T09:32:44.508 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:44 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-09-17T09:32:44.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: pgmap v203: 161 pgs: 18 active+undersized, 12 active+undersized+degraded, 131 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s; 72/999 objects degraded (7.207%); 0 B/s, 3 objects/s recovering 2024-09-17T09:32:44.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:44.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:44.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:44.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:32:44.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:32:44.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: pgmap v204: 161 pgs: 18 active+undersized, 12 active+undersized+degraded, 131 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s; 72/999 objects degraded (7.207%); 0 B/s, 4 objects/s recovering 2024-09-17T09:32:44.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: pgmap v205: 161 pgs: 18 active+undersized, 12 active+undersized+degraded, 131 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 44 op/s; 72/999 objects degraded (7.207%); 0 B/s, 4 objects/s recovering 2024-09-17T09:32:44.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:44.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:32:44.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:44.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:44.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:44.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:44.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: Upgrade: Setting container_image for all osd 2024-09-17T09:32:44.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:44.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-17T09:32:44.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-17T09:32:44.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-17T09:32:44.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-17T09:32:44.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-17T09:32:44.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-17T09:32:44.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-17T09:32:44.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-17T09:32:44.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: pgmap v203: 161 pgs: 18 active+undersized, 12 active+undersized+degraded, 131 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 74 op/s; 72/999 objects degraded (7.207%); 0 B/s, 3 objects/s recovering 2024-09-17T09:32:44.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:44.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:44.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:44.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:32:44.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:32:44.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: pgmap v204: 161 pgs: 18 active+undersized, 12 active+undersized+degraded, 131 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 41 op/s; 72/999 objects degraded (7.207%); 0 B/s, 4 objects/s recovering 2024-09-17T09:32:44.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: pgmap v205: 161 pgs: 18 active+undersized, 12 active+undersized+degraded, 131 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 44 op/s; 72/999 objects degraded (7.207%); 0 B/s, 4 objects/s recovering 2024-09-17T09:32:44.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:44.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:32:44.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:44.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: Upgrade: Setting container_image for all osd 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-17T09:32:44.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-17T09:32:44.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-17T09:32:44.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-17T09:32:44.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-17T09:32:44.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-17T09:32:44.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-17T09:32:44.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-17T09:32:44.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-09-17T09:32:44.553 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-17T09:32:44.553 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-17T09:32:44.553 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-17T09:32:44.553 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-17T09:32:44.553 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-17T09:32:44.554 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-17T09:32:44.554 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-17T09:32:44.554 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-17T09:32:44.554 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:44 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-09-17T09:32:45.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:45 smithi029 ceph-mon[58326]: Upgrade: Setting require_osd_release to 18 reef 2024-09-17T09:32:45.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:45 smithi029 ceph-mon[58326]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2024-09-17T09:32:45.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:45 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-09-17T09:32:45.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:45 smithi029 ceph-mon[58326]: osdmap e122: 8 total, 8 up, 8 in 2024-09-17T09:32:45.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:45 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:45.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:45 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:45.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:45 smithi029 ceph-mon[60489]: Upgrade: Setting require_osd_release to 18 reef 2024-09-17T09:32:45.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:45 smithi029 ceph-mon[60489]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2024-09-17T09:32:45.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:45 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-09-17T09:32:45.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:45 smithi029 ceph-mon[60489]: osdmap e122: 8 total, 8 up, 8 in 2024-09-17T09:32:45.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:45 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:45.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:45 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:45.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:45 smithi103 ceph-mon[49967]: Upgrade: Setting require_osd_release to 18 reef 2024-09-17T09:32:45.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:45 smithi103 ceph-mon[49967]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running quincy or later but require_osd_release < quincy) 2024-09-17T09:32:45.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:45 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-09-17T09:32:45.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:45 smithi103 ceph-mon[49967]: osdmap e122: 8 total, 8 up, 8 in 2024-09-17T09:32:45.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:45 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:45.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:45 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:46.379 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:46 smithi029 ceph-mon[60489]: Upgrade: Setting container_image for all mds 2024-09-17T09:32:46.379 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:46 smithi029 ceph-mon[60489]: pgmap v207: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 17 op/s; 0 B/s, 11 objects/s recovering 2024-09-17T09:32:46.379 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:46.380 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi029.tyykln", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:32:46.380 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:46.380 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:46 smithi029 ceph-mon[58326]: Upgrade: Setting container_image for all mds 2024-09-17T09:32:46.380 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:46 smithi029 ceph-mon[58326]: pgmap v207: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 17 op/s; 0 B/s, 11 objects/s recovering 2024-09-17T09:32:46.380 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:46.380 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi029.tyykln", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:32:46.380 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:46.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:46 smithi103 ceph-mon[49967]: Upgrade: Setting container_image for all mds 2024-09-17T09:32:46.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:46 smithi103 ceph-mon[49967]: pgmap v207: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 17 op/s; 0 B/s, 11 objects/s recovering 2024-09-17T09:32:46.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:46.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi029.tyykln", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:32:46.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:47.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:47 smithi029 ceph-mon[58326]: Upgrade: Updating rgw.foo.smithi029.tyykln (1/4) 2024-09-17T09:32:47.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:47 smithi029 ceph-mon[58326]: Deploying daemon rgw.foo.smithi029.tyykln on smithi029 2024-09-17T09:32:47.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:47 smithi029 ceph-mon[58326]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 72/999 objects degraded (7.207%), 12 pgs degraded) 2024-09-17T09:32:47.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:47 smithi029 ceph-mon[60489]: Upgrade: Updating rgw.foo.smithi029.tyykln (1/4) 2024-09-17T09:32:47.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:47 smithi029 ceph-mon[60489]: Deploying daemon rgw.foo.smithi029.tyykln on smithi029 2024-09-17T09:32:47.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:47 smithi029 ceph-mon[60489]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 72/999 objects degraded (7.207%), 12 pgs degraded) 2024-09-17T09:32:47.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:47 smithi103 ceph-mon[49967]: Upgrade: Updating rgw.foo.smithi029.tyykln (1/4) 2024-09-17T09:32:47.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:47 smithi103 ceph-mon[49967]: Deploying daemon rgw.foo.smithi029.tyykln on smithi029 2024-09-17T09:32:47.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:47 smithi103 ceph-mon[49967]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 72/999 objects degraded (7.207%), 12 pgs degraded) 2024-09-17T09:32:48.179 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:48 smithi029 ceph-mon[60489]: pgmap v208: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 12 op/s; 0 B/s, 8 objects/s recovering 2024-09-17T09:32:48.179 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:48 smithi029 ceph-mon[58326]: pgmap v208: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 12 op/s; 0 B/s, 8 objects/s recovering 2024-09-17T09:32:48.406 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:48 smithi103 ceph-mon[49967]: pgmap v208: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 12 op/s; 0 B/s, 8 objects/s recovering 2024-09-17T09:32:50.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:50 smithi103 ceph-mon[49967]: pgmap v209: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s; 0 B/s, 4 objects/s recovering 2024-09-17T09:32:50.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:50 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:50.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:50 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:50.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:50 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:50.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:50 smithi029 ceph-mon[60489]: pgmap v209: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s; 0 B/s, 4 objects/s recovering 2024-09-17T09:32:50.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:50 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:50.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:50 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:50.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:50 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:50 smithi029 ceph-mon[58326]: pgmap v209: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s; 0 B/s, 4 objects/s recovering 2024-09-17T09:32:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:50 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:50 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:50.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:50 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:52.201 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:52 smithi029 ceph-mon[58326]: pgmap v210: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 0 B/s, 3 objects/s recovering 2024-09-17T09:32:52.201 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:52 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:52.201 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:52 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi029.prstdw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:32:52.201 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:52 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:52.202 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:52 smithi029 ceph-mon[60489]: pgmap v210: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 0 B/s, 3 objects/s recovering 2024-09-17T09:32:52.202 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:52 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:52.202 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:52 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi029.prstdw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:32:52.202 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:52 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:52.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:52 smithi103 ceph-mon[49967]: pgmap v210: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 0 B/s, 3 objects/s recovering 2024-09-17T09:32:52.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:52 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:52.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:52 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi029.prstdw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:32:52.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:52 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:53.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:53 smithi029 ceph-mon[58326]: Upgrade: Updating rgw.smpl.smithi029.prstdw (2/4) 2024-09-17T09:32:53.243 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:53 smithi029 ceph-mon[58326]: Deploying daemon rgw.smpl.smithi029.prstdw on smithi029 2024-09-17T09:32:53.244 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:53 smithi029 ceph-mon[60489]: Upgrade: Updating rgw.smpl.smithi029.prstdw (2/4) 2024-09-17T09:32:53.244 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:53 smithi029 ceph-mon[60489]: Deploying daemon rgw.smpl.smithi029.prstdw on smithi029 2024-09-17T09:32:53.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:53 smithi103 ceph-mon[49967]: Upgrade: Updating rgw.smpl.smithi029.prstdw (2/4) 2024-09-17T09:32:53.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:53 smithi103 ceph-mon[49967]: Deploying daemon rgw.smpl.smithi029.prstdw on smithi029 2024-09-17T09:32:54.239 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:54 smithi029 ceph-mon[58326]: pgmap v211: 161 pgs: 161 active+clean; 604 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 204 B/s wr, 87 op/s; 0 B/s, 3 objects/s recovering 2024-09-17T09:32:54.240 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:54 smithi029 ceph-mon[60489]: pgmap v211: 161 pgs: 161 active+clean; 604 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 204 B/s wr, 87 op/s; 0 B/s, 3 objects/s recovering 2024-09-17T09:32:54.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:54 smithi103 ceph-mon[49967]: pgmap v211: 161 pgs: 161 active+clean; 604 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 204 B/s wr, 87 op/s; 0 B/s, 3 objects/s recovering 2024-09-17T09:32:55.566 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:32:56.149 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:56 smithi103 ceph-mon[49967]: pgmap v212: 161 pgs: 161 active+clean; 604 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 189 B/s wr, 115 op/s 2024-09-17T09:32:56.149 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:56 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:56.149 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:56 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:56.256 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:32:56.256 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (6m) 2m ago 11m 19.1M - 0.20.0 0881eb8f169f 7bd158a6d65c 2024-09-17T09:32:56.257 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (6m) 17s ago 10m 36.9M - 6.7.4 557c83e11646 e361816f7c14 2024-09-17T09:32:56.257 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 running (2m) 2m ago 10m - - 24ecd6d5f14c 8d25644a0887 2024-09-17T09:32:56.257 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (5m) 17s ago 15m 438M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:32:56.257 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (5m) 2m ago 16m 485M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:32:56.257 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (5m) 2m ago 16m 49.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:32:56.257 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (4m) 17s ago 15m 42.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 da050285f124 2024-09-17T09:32:56.257 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (4m) 2m ago 15m 45.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 0d3d1a65e2c9 2024-09-17T09:32:56.257 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (6m) 2m ago 11m 15.0M - 0.18.1 e5a616e4b9cf 34bfa263169f 2024-09-17T09:32:56.257 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (6m) 17s ago 11m 8866k - 0.18.1 e5a616e4b9cf 1adf1b5baad1 2024-09-17T09:32:56.257 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (3m) 2m ago 14m 52.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 4c456435000b 2024-09-17T09:32:56.257 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (2m) 2m ago 14m 44.3M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2ff250ddb1d9 2024-09-17T09:32:56.257 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (2m) 2m ago 14m 49.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7899f3c3c2f4 2024-09-17T09:32:56.257 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (2m) 2m ago 13m 12.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 94b84aca13ce 2024-09-17T09:32:56.258 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (119s) 17s ago 13m 49.3M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2306402255ea 2024-09-17T09:32:56.258 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (86s) 17s ago 13m 46.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0312d071d90b 2024-09-17T09:32:56.258 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (54s) 17s ago 12m 54.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 a5355c784b3c 2024-09-17T09:32:56.258 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (21s) 17s ago 12m 11.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 02f285335edf 2024-09-17T09:32:56.258 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 error 17s ago 12m - - 2024-09-17T09:32:56.258 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 starting - - - - 2024-09-17T09:32:56.258 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (10m) 17s ago 10m 41.9M - 16.2.0 24ecd6d5f14c 41637bf95184 2024-09-17T09:32:56.258 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 starting - - - - 2024-09-17T09:32:56.258 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (10m) 17s ago 10m 39.4M - 16.2.0 24ecd6d5f14c c59b7b6817d6 2024-09-17T09:32:56.546 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:56 smithi029 ceph-mon[60489]: pgmap v212: 161 pgs: 161 active+clean; 604 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 189 B/s wr, 115 op/s 2024-09-17T09:32:56.546 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:56 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:56.546 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:56 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:56.546 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:56 smithi029 ceph-mon[58326]: pgmap v212: 161 pgs: 161 active+clean; 604 KiB data, 299 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 189 B/s wr, 115 op/s 2024-09-17T09:32:56.546 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:56 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:56.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:56 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:56.660 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:32:56.660 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:32:56.660 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:32:56.660 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:32:56.661 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:32:56.661 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:32:56.661 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:32:56.661 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:32:56.661 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T09:32:56.661 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:32:56.661 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:32:56.661 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2, 2024-09-17T09:32:56.661 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 1 2024-09-17T09:32:56.661 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:32:56.661 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:32:56.661 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2, 2024-09-17T09:32:56.661 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 14 2024-09-17T09:32:56.662 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:32:56.662 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:32:57.000 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:32:57.001 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:32:57.001 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:32:57.001 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:32:57.001 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [ 2024-09-17T09:32:57.001 INFO:teuthology.orchestra.run.smithi029.stdout: "mon", 2024-09-17T09:32:57.001 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr", 2024-09-17T09:32:57.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd" 2024-09-17T09:32:57.001 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:32:57.001 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "13/23 daemons upgraded", 2024-09-17T09:32:57.001 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Currently upgrading rgw daemons", 2024-09-17T09:32:57.002 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:32:57.002 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:32:57.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:57 smithi103 ceph-mon[49967]: from='client.44925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:57.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:57 smithi103 ceph-mon[49967]: from='client.54832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:57.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:57 smithi103 ceph-mon[49967]: from='client.44964 172.21.15.29:0/2559851543' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:57.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:57 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:57.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:57 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi103.xonepa", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:32:57.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:57 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:57.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:57 smithi029 ceph-mon[60489]: from='client.44925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:57.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:57 smithi029 ceph-mon[60489]: from='client.54832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:57.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:57 smithi029 ceph-mon[60489]: from='client.44964 172.21.15.29:0/2559851543' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:57.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:57 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:57.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:57 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi103.xonepa", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:32:57.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:57 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:57.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:57 smithi029 ceph-mon[58326]: from='client.44925 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:57.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:57 smithi029 ceph-mon[58326]: from='client.54832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:57.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:57 smithi029 ceph-mon[58326]: from='client.44964 172.21.15.29:0/2559851543' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:32:57.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:57 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:32:57.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:57 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi103.xonepa", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:32:57.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:57 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:32:57.402 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T09:32:57.402 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T09:32:57.403 INFO:teuthology.orchestra.run.smithi029.stdout: daemon prometheus.a on smithi103 is in error state 2024-09-17T09:32:58.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:58 smithi103 ceph-mon[49967]: from='client.44958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:58.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:58 smithi103 ceph-mon[49967]: Upgrade: Updating rgw.foo.smithi103.xonepa (3/4) 2024-09-17T09:32:58.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:58 smithi103 ceph-mon[49967]: Deploying daemon rgw.foo.smithi103.xonepa on smithi103 2024-09-17T09:32:58.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:58 smithi103 ceph-mon[49967]: from='client.44970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:58.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:58 smithi103 ceph-mon[49967]: pgmap v213: 161 pgs: 161 active+clean; 604 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 255 B/s wr, 158 op/s 2024-09-17T09:32:58.219 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:32:58 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/2206336800' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:32:58.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:58 smithi029 ceph-mon[58326]: from='client.44958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:58.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:58 smithi029 ceph-mon[58326]: Upgrade: Updating rgw.foo.smithi103.xonepa (3/4) 2024-09-17T09:32:58.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:58 smithi029 ceph-mon[58326]: Deploying daemon rgw.foo.smithi103.xonepa on smithi103 2024-09-17T09:32:58.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:58 smithi029 ceph-mon[58326]: from='client.44970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:58.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:58 smithi029 ceph-mon[58326]: pgmap v213: 161 pgs: 161 active+clean; 604 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 255 B/s wr, 158 op/s 2024-09-17T09:32:58.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:32:58 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/2206336800' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:32:58.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:58 smithi029 ceph-mon[60489]: from='client.44958 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:58.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:58 smithi029 ceph-mon[60489]: Upgrade: Updating rgw.foo.smithi103.xonepa (3/4) 2024-09-17T09:32:58.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:58 smithi029 ceph-mon[60489]: Deploying daemon rgw.foo.smithi103.xonepa on smithi103 2024-09-17T09:32:58.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:58 smithi029 ceph-mon[60489]: from='client.44970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:32:58.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:58 smithi029 ceph-mon[60489]: pgmap v213: 161 pgs: 161 active+clean; 604 KiB data, 303 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 255 B/s wr, 158 op/s 2024-09-17T09:32:58.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:32:58 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/2206336800' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:33:00.329 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:00 smithi103 ceph-mon[49967]: pgmap v214: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 255 B/s wr, 186 op/s 2024-09-17T09:33:00.330 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1c"}]: dispatch 2024-09-17T09:33:00.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:00 smithi029 ceph-mon[58326]: pgmap v214: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 255 B/s wr, 186 op/s 2024-09-17T09:33:00.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1c"}]: dispatch 2024-09-17T09:33:00.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:00 smithi029 ceph-mon[60489]: pgmap v214: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 255 B/s wr, 186 op/s 2024-09-17T09:33:00.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1c"}]: dispatch 2024-09-17T09:33:01.265 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1c"}]': finished 2024-09-17T09:33:01.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:01 smithi103 ceph-mon[49967]: osdmap e123: 8 total, 8 up, 8 in 2024-09-17T09:33:01.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:01.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:01 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:01.266 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:01 smithi103 ceph-mon[49967]: osdmap e124: 8 total, 8 up, 8 in 2024-09-17T09:33:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1c"}]': finished 2024-09-17T09:33:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:01 smithi029 ceph-mon[58326]: osdmap e123: 8 total, 8 up, 8 in 2024-09-17T09:33:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:01 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:01.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:01 smithi029 ceph-mon[58326]: osdmap e124: 8 total, 8 up, 8 in 2024-09-17T09:33:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1c"}]': finished 2024-09-17T09:33:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:01 smithi029 ceph-mon[60489]: osdmap e123: 8 total, 8 up, 8 in 2024-09-17T09:33:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:01 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:01.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:01 smithi029 ceph-mon[60489]: osdmap e124: 8 total, 8 up, 8 in 2024-09-17T09:33:02.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:02 smithi103 ceph-mon[49967]: pgmap v216: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 307 B/s wr, 221 op/s 2024-09-17T09:33:02.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:02 smithi029 ceph-mon[58326]: pgmap v216: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 307 B/s wr, 221 op/s 2024-09-17T09:33:02.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:02 smithi029 ceph-mon[60489]: pgmap v216: 161 pgs: 161 active+clean; 604 KiB data, 307 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 307 B/s wr, 221 op/s 2024-09-17T09:33:03.561 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:03 smithi103 ceph-mon[49967]: Upgrade: Updating rgw.smpl.smithi103.gwwbkl (4/4) 2024-09-17T09:33:03.562 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:03 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:03.562 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:03 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi103.gwwbkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:33:03.562 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:03 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:03.562 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:03 smithi103 ceph-mon[49967]: Deploying daemon rgw.smpl.smithi103.gwwbkl on smithi103 2024-09-17T09:33:03.562 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:03 smithi103 ceph-mon[49967]: pgmap v218: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 255 B/s wr, 206 op/s 2024-09-17T09:33:03.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:03 smithi029 ceph-mon[58326]: Upgrade: Updating rgw.smpl.smithi103.gwwbkl (4/4) 2024-09-17T09:33:03.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:03 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:03.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:03 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi103.gwwbkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:33:03.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:03 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:03.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:03 smithi029 ceph-mon[58326]: Deploying daemon rgw.smpl.smithi103.gwwbkl on smithi103 2024-09-17T09:33:03.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:03 smithi029 ceph-mon[58326]: pgmap v218: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 255 B/s wr, 206 op/s 2024-09-17T09:33:03.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:03 smithi029 ceph-mon[60489]: Upgrade: Updating rgw.smpl.smithi103.gwwbkl (4/4) 2024-09-17T09:33:03.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:03 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:03.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:03 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi103.gwwbkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T09:33:03.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:03 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:03.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:03 smithi029 ceph-mon[60489]: Deploying daemon rgw.smpl.smithi103.gwwbkl on smithi103 2024-09-17T09:33:03.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:03 smithi029 ceph-mon[60489]: pgmap v218: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 255 B/s wr, 206 op/s 2024-09-17T09:33:06.477 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:06 smithi103 ceph-mon[49967]: pgmap v219: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 127 B/s wr, 166 op/s 2024-09-17T09:33:06.477 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:06 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:06.477 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:06 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:06.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:06 smithi029 ceph-mon[58326]: pgmap v219: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 127 B/s wr, 166 op/s 2024-09-17T09:33:06.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:06 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:06.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:06 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:06.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:06 smithi029 ceph-mon[60489]: pgmap v219: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 127 B/s wr, 166 op/s 2024-09-17T09:33:06.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:06 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:06.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:06 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:08.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:08 smithi103 ceph-mon[49967]: pgmap v220: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 255 B/s wr, 202 op/s 2024-09-17T09:33:08.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:08 smithi029 ceph-mon[58326]: pgmap v220: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 255 B/s wr, 202 op/s 2024-09-17T09:33:08.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:08 smithi029 ceph-mon[60489]: pgmap v220: 161 pgs: 161 active+clean; 604 KiB data, 311 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 255 B/s wr, 202 op/s 2024-09-17T09:33:10.244 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:10 smithi103 ceph-mon[49967]: pgmap v221: 161 pgs: 161 active+clean; 604 KiB data, 315 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 228 B/s wr, 218 op/s 2024-09-17T09:33:10.244 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:10 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.244 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:10 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.244 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:10 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.245 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:10 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.245 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:10 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.245 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:10 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.245 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:10 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.245 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:10 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[58326]: pgmap v221: 161 pgs: 161 active+clean; 604 KiB data, 315 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 228 B/s wr, 218 op/s 2024-09-17T09:33:10.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.453 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.454 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.454 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.454 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.454 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[60489]: pgmap v221: 161 pgs: 161 active+clean; 604 KiB data, 315 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 228 B/s wr, 218 op/s 2024-09-17T09:33:10.454 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.454 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.454 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.454 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.454 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.455 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.455 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:10.455 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:10 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:12.385 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:12 smithi029 ceph-mon[60489]: pgmap v222: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 204 B/s wr, 185 op/s 2024-09-17T09:33:12.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:12 smithi029 ceph-mon[58326]: pgmap v222: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 204 B/s wr, 185 op/s 2024-09-17T09:33:12.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:12 smithi103 ceph-mon[49967]: pgmap v222: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 204 B/s wr, 185 op/s 2024-09-17T09:33:14.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: Detected new or changed devices on smithi103 2024-09-17T09:33:14.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: pgmap v223: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 170 B/s wr, 155 op/s 2024-09-17T09:33:14.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: Detected new or changed devices on smithi029 2024-09-17T09:33:14.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: pgmap v224: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 102 B/s wr, 133 op/s 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: pgmap v225: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 127 B/s wr, 124 op/s 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.257 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi029.tyykln"}]: dispatch 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi029.tyykln"}]': finished 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi103.xonepa"}]: dispatch 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi103.xonepa"}]': finished 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi029.prstdw"}]: dispatch 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi029.prstdw"}]': finished 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi103.gwwbkl"}]: dispatch 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi103.gwwbkl"}]': finished 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.258 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:13 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: Detected new or changed devices on smithi103 2024-09-17T09:33:14.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: pgmap v223: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 170 B/s wr, 155 op/s 2024-09-17T09:33:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: Detected new or changed devices on smithi029 2024-09-17T09:33:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:33:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:33:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: pgmap v224: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 102 B/s wr, 133 op/s 2024-09-17T09:33:14.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: pgmap v225: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 127 B/s wr, 124 op/s 2024-09-17T09:33:14.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:33:14.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi029.tyykln"}]: dispatch 2024-09-17T09:33:14.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi029.tyykln"}]': finished 2024-09-17T09:33:14.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi103.xonepa"}]: dispatch 2024-09-17T09:33:14.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi103.xonepa"}]': finished 2024-09-17T09:33:14.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi029.prstdw"}]: dispatch 2024-09-17T09:33:14.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: Detected new or changed devices on smithi103 2024-09-17T09:33:14.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: pgmap v223: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 170 B/s wr, 155 op/s 2024-09-17T09:33:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: Detected new or changed devices on smithi029 2024-09-17T09:33:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:33:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:33:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: pgmap v224: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 102 B/s wr, 133 op/s 2024-09-17T09:33:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: pgmap v225: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 127 B/s wr, 124 op/s 2024-09-17T09:33:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:33:14.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi029.tyykln"}]: dispatch 2024-09-17T09:33:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi029.tyykln"}]': finished 2024-09-17T09:33:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi103.xonepa"}]: dispatch 2024-09-17T09:33:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi103.xonepa"}]': finished 2024-09-17T09:33:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi029.prstdw"}]: dispatch 2024-09-17T09:33:14.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi029.prstdw"}]': finished 2024-09-17T09:33:14.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi103.gwwbkl"}]: dispatch 2024-09-17T09:33:14.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi103.gwwbkl"}]': finished 2024-09-17T09:33:14.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi029.prstdw"}]': finished 2024-09-17T09:33:14.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi103.gwwbkl"}]: dispatch 2024-09-17T09:33:14.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi103.gwwbkl"}]': finished 2024-09-17T09:33:14.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:14.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:14.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:13 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[58326]: Upgrade: Setting container_image for all rgw 2024-09-17T09:33:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[58326]: Upgrade: Setting container_image for all rbd-mirror 2024-09-17T09:33:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[58326]: Upgrade: Setting container_image for all ceph-exporter 2024-09-17T09:33:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[58326]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:33:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:33:15.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:15.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[60489]: Upgrade: Setting container_image for all rgw 2024-09-17T09:33:15.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[60489]: Upgrade: Setting container_image for all rbd-mirror 2024-09-17T09:33:15.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[60489]: Upgrade: Setting container_image for all ceph-exporter 2024-09-17T09:33:15.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[60489]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:33:15.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:15.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:33:15.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:15.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:14 smithi103 ceph-mon[49967]: Upgrade: Setting container_image for all rgw 2024-09-17T09:33:15.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:14 smithi103 ceph-mon[49967]: Upgrade: Setting container_image for all rbd-mirror 2024-09-17T09:33:15.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:14 smithi103 ceph-mon[49967]: Upgrade: Setting container_image for all ceph-exporter 2024-09-17T09:33:15.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:14 smithi103 ceph-mon[49967]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:33:15.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:15.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi029.gellkh", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T09:33:15.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:16.151 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:15 smithi029 ceph-mon[60489]: Upgrade: Updating iscsi.foo.smithi029.gellkh 2024-09-17T09:33:16.151 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:15 smithi029 ceph-mon[60489]: Deploying daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:33:16.151 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:15 smithi029 ceph-mon[60489]: pgmap v226: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 45 op/s 2024-09-17T09:33:16.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:15 smithi029 ceph-mon[58326]: Upgrade: Updating iscsi.foo.smithi029.gellkh 2024-09-17T09:33:16.152 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:15 smithi029 ceph-mon[58326]: Deploying daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:33:16.152 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:15 smithi029 ceph-mon[58326]: pgmap v226: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 45 op/s 2024-09-17T09:33:16.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:15 smithi103 ceph-mon[49967]: Upgrade: Updating iscsi.foo.smithi029.gellkh 2024-09-17T09:33:16.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:15 smithi103 ceph-mon[49967]: Deploying daemon iscsi.foo.smithi029.gellkh on smithi029 2024-09-17T09:33:16.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:15 smithi103 ceph-mon[49967]: pgmap v226: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 45 op/s 2024-09-17T09:33:18.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:18 smithi103 ceph-mon[49967]: pgmap v227: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-17T09:33:18.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:18 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:18.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:18 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:18.525 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:18 smithi029 ceph-mon[58326]: pgmap v227: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-17T09:33:18.525 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:18 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:18.525 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:18 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:18.526 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:18 smithi029 ceph-mon[60489]: pgmap v227: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-17T09:33:18.526 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:18 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:18.526 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:18 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:19.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:19 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/3656687520' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T09:33:19.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:19 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/1072745415' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2438855422"}]: dispatch 2024-09-17T09:33:19.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:19 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2438855422"}]: dispatch 2024-09-17T09:33:19.485 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:19 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/3656687520' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T09:33:19.485 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:19 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/1072745415' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2438855422"}]: dispatch 2024-09-17T09:33:19.485 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:19 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2438855422"}]: dispatch 2024-09-17T09:33:19.486 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:19 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/3656687520' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T09:33:19.486 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:19 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/1072745415' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2438855422"}]: dispatch 2024-09-17T09:33:19.486 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:19 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2438855422"}]: dispatch 2024-09-17T09:33:20.491 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:20 smithi103 ceph-mon[49967]: pgmap v228: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:33:20.491 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:20 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2438855422"}]': finished 2024-09-17T09:33:20.491 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:20 smithi103 ceph-mon[49967]: osdmap e125: 8 total, 8 up, 8 in 2024-09-17T09:33:20.491 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:20 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/3384466957' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: dispatch 2024-09-17T09:33:20.546 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:20 smithi029 ceph-mon[60489]: pgmap v228: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:33:20.546 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:20 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2438855422"}]': finished 2024-09-17T09:33:20.546 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:20 smithi029 ceph-mon[60489]: osdmap e125: 8 total, 8 up, 8 in 2024-09-17T09:33:20.546 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:20 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/3384466957' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: dispatch 2024-09-17T09:33:20.546 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:20 smithi029 ceph-mon[58326]: pgmap v228: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:33:20.546 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:20 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2438855422"}]': finished 2024-09-17T09:33:20.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:20 smithi029 ceph-mon[58326]: osdmap e125: 8 total, 8 up, 8 in 2024-09-17T09:33:20.547 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:20 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/3384466957' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]: dispatch 2024-09-17T09:33:21.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:21 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/3384466957' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]': finished 2024-09-17T09:33:21.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:21 smithi103 ceph-mon[49967]: osdmap e126: 8 total, 8 up, 8 in 2024-09-17T09:33:21.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:21 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/474568276' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/688662900"}]: dispatch 2024-09-17T09:33:21.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:21 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/688662900"}]: dispatch 2024-09-17T09:33:21.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:21 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:21.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:21 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:21.510 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:21 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/3384466957' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]': finished 2024-09-17T09:33:21.510 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:21 smithi029 ceph-mon[58326]: osdmap e126: 8 total, 8 up, 8 in 2024-09-17T09:33:21.510 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:21 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/474568276' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/688662900"}]: dispatch 2024-09-17T09:33:21.510 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:21 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/688662900"}]: dispatch 2024-09-17T09:33:21.510 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:21 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:21.510 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:21 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:21.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:21 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/3384466957' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/629296380"}]': finished 2024-09-17T09:33:21.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:21 smithi029 ceph-mon[60489]: osdmap e126: 8 total, 8 up, 8 in 2024-09-17T09:33:21.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:21 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/474568276' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/688662900"}]: dispatch 2024-09-17T09:33:21.510 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:21 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/688662900"}]: dispatch 2024-09-17T09:33:21.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:21 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:21.511 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:21 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:22.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:22 smithi103 ceph-mon[49967]: pgmap v231: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:33:22.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:22 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/688662900"}]': finished 2024-09-17T09:33:22.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:22 smithi103 ceph-mon[49967]: osdmap e127: 8 total, 8 up, 8 in 2024-09-17T09:33:22.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:22 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/3401801519' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/629296380"}]: dispatch 2024-09-17T09:33:22.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:22 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/629296380"}]: dispatch 2024-09-17T09:33:22.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:22 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:22.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:22 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:22.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[58326]: pgmap v231: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:33:22.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/688662900"}]': finished 2024-09-17T09:33:22.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[58326]: osdmap e127: 8 total, 8 up, 8 in 2024-09-17T09:33:22.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/3401801519' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/629296380"}]: dispatch 2024-09-17T09:33:22.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/629296380"}]: dispatch 2024-09-17T09:33:22.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:22.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:22.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[60489]: pgmap v231: 161 pgs: 161 active+clean; 604 KiB data, 319 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:33:22.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/688662900"}]': finished 2024-09-17T09:33:22.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[60489]: osdmap e127: 8 total, 8 up, 8 in 2024-09-17T09:33:22.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/3401801519' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/629296380"}]: dispatch 2024-09-17T09:33:22.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/629296380"}]: dispatch 2024-09-17T09:33:22.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:22.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:22 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:23.486 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:23 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/629296380"}]': finished 2024-09-17T09:33:23.486 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:23 smithi029 ceph-mon[60489]: osdmap e128: 8 total, 8 up, 8 in 2024-09-17T09:33:23.486 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:23 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:23.486 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:23 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:23.486 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:23 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/357458947' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: dispatch 2024-09-17T09:33:23.487 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:23 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/629296380"}]': finished 2024-09-17T09:33:23.487 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:23 smithi029 ceph-mon[58326]: osdmap e128: 8 total, 8 up, 8 in 2024-09-17T09:33:23.487 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:23 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:23.487 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:23 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:23.487 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:23 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/357458947' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: dispatch 2024-09-17T09:33:23.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:23 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/629296380"}]': finished 2024-09-17T09:33:23.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:23 smithi103 ceph-mon[49967]: osdmap e128: 8 total, 8 up, 8 in 2024-09-17T09:33:23.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:23 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:23.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:23 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:23.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:23 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/357458947' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]: dispatch 2024-09-17T09:33:24.488 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:24 smithi029 ceph-mon[58326]: pgmap v234: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:24.488 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:24 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/357458947' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]': finished 2024-09-17T09:33:24.488 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:24 smithi029 ceph-mon[58326]: osdmap e129: 8 total, 8 up, 8 in 2024-09-17T09:33:24.488 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:24 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/3384091588' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/1475002192"}]: dispatch 2024-09-17T09:33:24.488 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:24 smithi029 ceph-mon[60489]: pgmap v234: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:24.488 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:24 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/357458947' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]': finished 2024-09-17T09:33:24.489 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:24 smithi029 ceph-mon[60489]: osdmap e129: 8 total, 8 up, 8 in 2024-09-17T09:33:24.489 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:24 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/3384091588' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/1475002192"}]: dispatch 2024-09-17T09:33:24.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:24 smithi103 ceph-mon[49967]: pgmap v234: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:24 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/357458947' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/1161843867"}]': finished 2024-09-17T09:33:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:24 smithi103 ceph-mon[49967]: osdmap e129: 8 total, 8 up, 8 in 2024-09-17T09:33:24.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:24 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/3384091588' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/1475002192"}]: dispatch 2024-09-17T09:33:25.711 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:25 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/3384091588' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/1475002192"}]': finished 2024-09-17T09:33:25.711 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:25 smithi029 ceph-mon[60489]: osdmap e130: 8 total, 8 up, 8 in 2024-09-17T09:33:25.711 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:25 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/2997712757' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/536255971"}]: dispatch 2024-09-17T09:33:25.711 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:25 smithi029 ceph-mon[60489]: pgmap v237: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:25 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/3384091588' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/1475002192"}]': finished 2024-09-17T09:33:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:25 smithi029 ceph-mon[58326]: osdmap e130: 8 total, 8 up, 8 in 2024-09-17T09:33:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:25 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/2997712757' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/536255971"}]: dispatch 2024-09-17T09:33:25.712 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:25 smithi029 ceph-mon[58326]: pgmap v237: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:25 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/3384091588' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/1475002192"}]': finished 2024-09-17T09:33:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:25 smithi103 ceph-mon[49967]: osdmap e130: 8 total, 8 up, 8 in 2024-09-17T09:33:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:25 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/2997712757' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/536255971"}]: dispatch 2024-09-17T09:33:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:25 smithi103 ceph-mon[49967]: pgmap v237: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/2997712757' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/536255971"}]': finished 2024-09-17T09:33:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: osdmap e131: 8 total, 8 up, 8 in 2024-09-17T09:33:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:26.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:33:26.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:33:26.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: pgmap v239: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:33:26.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: pgmap v240: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:33:26.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: Checking dashboard <-> RGW credentials 2024-09-17T09:33:26.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/2568226091' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3878824021"}]: dispatch 2024-09-17T09:33:26.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3878824021"}]: dispatch 2024-09-17T09:33:26.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:33:26.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: Upgrade: Setting container_image for all iscsi 2024-09-17T09:33:26.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi029.gellkh"}]: dispatch 2024-09-17T09:33:26.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi029.gellkh"}]': finished 2024-09-17T09:33:26.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: Upgrade: Setting container_image for all nfs 2024-09-17T09:33:26.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: Upgrade: Setting container_image for all nvmeof 2024-09-17T09:33:26.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:26 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/2997712757' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/536255971"}]': finished 2024-09-17T09:33:26.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: osdmap e131: 8 total, 8 up, 8 in 2024-09-17T09:33:26.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:26.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:33:26.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:33:26.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: pgmap v239: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:33:26.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: pgmap v240: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:33:26.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: Checking dashboard <-> RGW credentials 2024-09-17T09:33:26.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/2568226091' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3878824021"}]: dispatch 2024-09-17T09:33:26.761 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3878824021"}]: dispatch 2024-09-17T09:33:26.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:33:26.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:33:26.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:33:26.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:33:26.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:33:26.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.762 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.763 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.763 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.763 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/2997712757' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/536255971"}]': finished 2024-09-17T09:33:26.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: osdmap e131: 8 total, 8 up, 8 in 2024-09-17T09:33:26.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:26.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:33:26.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: pgmap v239: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: pgmap v240: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: Checking dashboard <-> RGW credentials 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/2568226091' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3878824021"}]: dispatch 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3878824021"}]: dispatch 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:33:26.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.765 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.765 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.765 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.765 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: Upgrade: Setting container_image for all iscsi 2024-09-17T09:33:26.766 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.766 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi029.gellkh"}]: dispatch 2024-09-17T09:33:26.766 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi029.gellkh"}]': finished 2024-09-17T09:33:26.766 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.766 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: Upgrade: Setting container_image for all nfs 2024-09-17T09:33:26.766 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.766 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.766 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: Upgrade: Setting container_image for all nvmeof 2024-09-17T09:33:26.766 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.766 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.766 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.766 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: Upgrade: Setting container_image for all iscsi 2024-09-17T09:33:26.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi029.gellkh"}]: dispatch 2024-09-17T09:33:26.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi.foo.smithi029.gellkh"}]': finished 2024-09-17T09:33:26.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: Upgrade: Setting container_image for all nfs 2024-09-17T09:33:26.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:26.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:26.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: Upgrade: Setting container_image for all nvmeof 2024-09-17T09:33:26.767 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:26 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:27.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:27 smithi103 ceph-mon[49967]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:33:27.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:27 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3878824021"}]': finished 2024-09-17T09:33:27.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:27 smithi103 ceph-mon[49967]: osdmap e132: 8 total, 8 up, 8 in 2024-09-17T09:33:27.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:27 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/1489606152' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: dispatch 2024-09-17T09:33:27.763 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:27 smithi029 ceph-mon[60489]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:33:27.763 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:27 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3878824021"}]': finished 2024-09-17T09:33:27.763 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:27 smithi029 ceph-mon[60489]: osdmap e132: 8 total, 8 up, 8 in 2024-09-17T09:33:27.763 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:27 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/1489606152' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: dispatch 2024-09-17T09:33:27.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:27 smithi029 ceph-mon[58326]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T09:33:27.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:27 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3878824021"}]': finished 2024-09-17T09:33:27.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:27 smithi029 ceph-mon[58326]: osdmap e132: 8 total, 8 up, 8 in 2024-09-17T09:33:27.764 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:27 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/1489606152' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/37491942"}]: dispatch 2024-09-17T09:33:27.767 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:33:28.298 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:28 smithi029 systemd[1]: Stopping Ceph node-exporter.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:33:28.298 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:28 smithi029 podman[87273]: 2024-09-17 09:33:28.248494013 +0000 UTC m=+0.141553120 container died 34bfa263169ff229618628f882c64f7d1fb220209c3d55de705b616ff310bdc5 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T09:33:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:33:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (7m) 6s ago 11m 19.1M - 0.20.0 0881eb8f169f 7bd158a6d65c 2024-09-17T09:33:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (6m) 7s ago 11m 36.9M - 6.7.4 557c83e11646 e361816f7c14 2024-09-17T09:33:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 running (11s) 6s ago 10m 64.0M - 3.8 980a0bfcd684 c5c886a0c37c 2024-09-17T09:33:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (5m) 7s ago 15m 438M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:33:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (6m) 6s ago 16m 492M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:33:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (5m) 6s ago 17m 58.4M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:33:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (4m) 7s ago 15m 45.1M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 da050285f124 2024-09-17T09:33:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (5m) 6s ago 16m 52.2M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 0d3d1a65e2c9 2024-09-17T09:33:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (7m) 6s ago 12m 15.1M - 0.18.1 e5a616e4b9cf 34bfa263169f 2024-09-17T09:33:28.469 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (6m) 7s ago 12m 8866k - 0.18.1 e5a616e4b9cf 1adf1b5baad1 2024-09-17T09:33:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (4m) 6s ago 15m 64.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 4c456435000b 2024-09-17T09:33:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (3m) 6s ago 14m 57.1M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2ff250ddb1d9 2024-09-17T09:33:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (3m) 6s ago 14m 58.7M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7899f3c3c2f4 2024-09-17T09:33:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (2m) 6s ago 14m 62.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 94b84aca13ce 2024-09-17T09:33:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (2m) 7s ago 14m 55.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2306402255ea 2024-09-17T09:33:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (119s) 7s ago 13m 49.9M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0312d071d90b 2024-09-17T09:33:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (86s) 7s ago 13m 59.7M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 a5355c784b3c 2024-09-17T09:33:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (53s) 7s ago 13m 49.2M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 02f285335edf 2024-09-17T09:33:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 error 7s ago 12m - - 2024-09-17T09:33:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (39s) 6s ago 11m 77.0M - 18.2.4-855-g19e2a2be 980a0bfcd684 697c12a1dd89 2024-09-17T09:33:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (28s) 7s ago 10m 73.6M - 18.2.4-855-g19e2a2be 980a0bfcd684 6b0ca358485b 2024-09-17T09:33:28.470 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (33s) 6s ago 10m 74.8M - 18.2.4-855-g19e2a2be 980a0bfcd684 1b791ec24040 2024-09-17T09:33:28.471 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (23s) 7s ago 10m 74.7M - 18.2.4-855-g19e2a2be 980a0bfcd684 ddcc313b787c 2024-09-17T09:33:28.751 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:28 smithi029 podman[87273]: 2024-09-17 09:33:28.647575264 +0000 UTC m=+0.540634371 container cleanup 34bfa263169ff229618628f882c64f7d1fb220209c3d55de705b616ff310bdc5 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T09:33:28.751 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:28 smithi029 bash[87273]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a 2024-09-17T09:33:28.878 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:33:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:33:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:33:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:33:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:33:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:33:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:33:28.878 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:33:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T09:33:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:33:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:33:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T09:33:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:33:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:33:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 17 2024-09-17T09:33:28.879 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:33:28.879 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:33:29.048 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:28 smithi029 podman[87308]: 2024-09-17 09:33:28.902029214 +0000 UTC m=+0.656667382 container remove 34bfa263169ff229618628f882c64f7d1fb220209c3d55de705b616ff310bdc5 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.a, maintainer=The Prometheus Authors ) 2024-09-17T09:33:29.048 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:28 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:33:29.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[58326]: Upgrade: Updating node-exporter.a (1/2) 2024-09-17T09:33:29.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[58326]: Deploying daemon node-exporter.a on smithi029 2024-09-17T09:33:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[58326]: pgmap v243: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 8.4 KiB/s rd, 782 B/s wr, 9 op/s 2024-09-17T09:33:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/1489606152' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/37491942"}]': finished 2024-09-17T09:33:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[58326]: osdmap e133: 8 total, 8 up, 8 in 2024-09-17T09:33:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[58326]: from='client.45147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:33:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:33:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/3052599851' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/340401167"}]: dispatch 2024-09-17T09:33:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/340401167"}]: dispatch 2024-09-17T09:33:29.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[58326]: from='client.45159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:33:29.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[60489]: Upgrade: Updating node-exporter.a (1/2) 2024-09-17T09:33:29.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[60489]: Deploying daemon node-exporter.a on smithi029 2024-09-17T09:33:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[60489]: pgmap v243: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 8.4 KiB/s rd, 782 B/s wr, 9 op/s 2024-09-17T09:33:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/1489606152' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/37491942"}]': finished 2024-09-17T09:33:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[60489]: osdmap e133: 8 total, 8 up, 8 in 2024-09-17T09:33:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[60489]: from='client.45147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:33:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:33:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/3052599851' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/340401167"}]: dispatch 2024-09-17T09:33:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/340401167"}]: dispatch 2024-09-17T09:33:29.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:28 smithi029 ceph-mon[60489]: from='client.45159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:33:29.213 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:28 smithi103 ceph-mon[49967]: Upgrade: Updating node-exporter.a (1/2) 2024-09-17T09:33:29.213 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:28 smithi103 ceph-mon[49967]: Deploying daemon node-exporter.a on smithi029 2024-09-17T09:33:29.214 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:28 smithi103 ceph-mon[49967]: pgmap v243: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 8.4 KiB/s rd, 782 B/s wr, 9 op/s 2024-09-17T09:33:29.214 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:28 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/1489606152' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/37491942"}]': finished 2024-09-17T09:33:29.214 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:28 smithi103 ceph-mon[49967]: osdmap e133: 8 total, 8 up, 8 in 2024-09-17T09:33:29.214 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:28 smithi103 ceph-mon[49967]: from='client.45147 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:33:29.214 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:28 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:33:29.215 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:28 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/3052599851' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/340401167"}]: dispatch 2024-09-17T09:33:29.215 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:28 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/340401167"}]: dispatch 2024-09-17T09:33:29.215 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:28 smithi103 ceph-mon[49967]: from='client.45159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:33:29.242 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:33:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:33:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:33:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:33:29.242 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [ 2024-09-17T09:33:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "osd", 2024-09-17T09:33:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr", 2024-09-17T09:33:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "iscsi", 2024-09-17T09:33:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "mon", 2024-09-17T09:33:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw" 2024-09-17T09:33:29.243 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:33:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "18/23 daemons upgraded", 2024-09-17T09:33:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Currently upgrading node-exporter daemons", 2024-09-17T09:33:29.244 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:33:29.244 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:33:29.548 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:29 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-17T09:33:29.548 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:29 smithi029 systemd[1]: Stopped Ceph node-exporter.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:33:29.548 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:29 smithi029 systemd[1]: Starting Ceph node-exporter.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:33:29.693 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T09:33:29.693 INFO:teuthology.orchestra.run.smithi029.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T09:33:29.693 INFO:teuthology.orchestra.run.smithi029.stdout: daemon prometheus.a on smithi103 is in error state 2024-09-17T09:33:29.967 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:29 smithi029 bash[87539]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-17T09:33:29.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[58326]: from='client.45165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:33:29.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/340401167"}]': finished 2024-09-17T09:33:29.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[58326]: osdmap e134: 8 total, 8 up, 8 in 2024-09-17T09:33:29.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[58326]: from='client.55054 172.21.15.29:0/1060529145' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:29.967 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/2407221417' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]: dispatch 2024-09-17T09:33:29.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]: dispatch 2024-09-17T09:33:29.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[58326]: from='client.45183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:33:29.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[58326]: from='client.55072 172.21.15.29:0/983212640' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:33:29.968 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[60489]: from='client.45165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:33:29.968 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/340401167"}]': finished 2024-09-17T09:33:29.968 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[60489]: osdmap e134: 8 total, 8 up, 8 in 2024-09-17T09:33:29.968 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[60489]: from='client.55054 172.21.15.29:0/1060529145' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:29.968 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/2407221417' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]: dispatch 2024-09-17T09:33:29.969 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]: dispatch 2024-09-17T09:33:29.969 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[60489]: from='client.45183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:33:29.969 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:29 smithi029 ceph-mon[60489]: from='client.55072 172.21.15.29:0/983212640' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:33:30.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:29 smithi103 ceph-mon[49967]: from='client.45165 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:33:30.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:29 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/340401167"}]': finished 2024-09-17T09:33:30.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:29 smithi103 ceph-mon[49967]: osdmap e134: 8 total, 8 up, 8 in 2024-09-17T09:33:30.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:29 smithi103 ceph-mon[49967]: from='client.55054 172.21.15.29:0/1060529145' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:30.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:29 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/2407221417' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]: dispatch 2024-09-17T09:33:30.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:29 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]: dispatch 2024-09-17T09:33:30.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:29 smithi103 ceph-mon[49967]: from='client.45183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:33:30.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:29 smithi103 ceph-mon[49967]: from='client.55072 172.21.15.29:0/983212640' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:33:30.298 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:29 smithi029 bash[87539]: Getting image source signatures 2024-09-17T09:33:30.298 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:29 smithi029 bash[87539]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-17T09:33:30.298 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:29 smithi029 bash[87539]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-17T09:33:30.298 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:29 smithi029 bash[87539]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-17T09:33:31.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:30 smithi029 ceph-mon[60489]: pgmap v245: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-17T09:33:31.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:30 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]': finished 2024-09-17T09:33:31.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:30 smithi029 ceph-mon[60489]: osdmap e135: 8 total, 8 up, 8 in 2024-09-17T09:33:31.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:30 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/253830286' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/4068330535"}]: dispatch 2024-09-17T09:33:31.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:30 smithi029 ceph-mon[58326]: pgmap v245: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-17T09:33:31.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:30 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]': finished 2024-09-17T09:33:31.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:30 smithi029 ceph-mon[58326]: osdmap e135: 8 total, 8 up, 8 in 2024-09-17T09:33:31.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:30 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/253830286' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/4068330535"}]: dispatch 2024-09-17T09:33:31.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:30 smithi103 ceph-mon[49967]: pgmap v245: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-17T09:33:31.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:30 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/4237070615"}]': finished 2024-09-17T09:33:31.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:30 smithi103 ceph-mon[49967]: osdmap e135: 8 total, 8 up, 8 in 2024-09-17T09:33:31.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:30 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/253830286' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/4068330535"}]: dispatch 2024-09-17T09:33:32.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:31 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/253830286' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/4068330535"}]': finished 2024-09-17T09:33:32.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:31 smithi029 ceph-mon[60489]: osdmap e136: 8 total, 8 up, 8 in 2024-09-17T09:33:32.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:31 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/1623172648' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/536255971"}]: dispatch 2024-09-17T09:33:32.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:31 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/536255971"}]: dispatch 2024-09-17T09:33:32.048 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:31 smithi029 bash[87539]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-17T09:33:32.048 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:31 smithi029 bash[87539]: Writing manifest to image destination 2024-09-17T09:33:32.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:31 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/253830286' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/4068330535"}]': finished 2024-09-17T09:33:32.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:31 smithi029 ceph-mon[58326]: osdmap e136: 8 total, 8 up, 8 in 2024-09-17T09:33:32.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:31 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/1623172648' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/536255971"}]: dispatch 2024-09-17T09:33:32.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:31 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/536255971"}]: dispatch 2024-09-17T09:33:32.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:31 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/253830286' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/4068330535"}]': finished 2024-09-17T09:33:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:31 smithi103 ceph-mon[49967]: osdmap e136: 8 total, 8 up, 8 in 2024-09-17T09:33:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:31 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/1623172648' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/536255971"}]: dispatch 2024-09-17T09:33:32.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:31 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/536255971"}]: dispatch 2024-09-17T09:33:32.392 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 podman[87539]: 2024-09-17 09:33:32.036684991 +0000 UTC m=+2.500005075 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-17T09:33:32.392 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 podman[87539]: 2024-09-17 09:33:32.161589129 +0000 UTC m=+2.624909212 container create 5f8842670a3c5e1e6e850c7abbe8958f2b6788df2b3f24cf28362ab8f79b9e03 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T09:33:32.775 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 podman[87539]: 2024-09-17 09:33:32.392052555 +0000 UTC m=+2.855372648 container init 5f8842670a3c5e1e6e850c7abbe8958f2b6788df2b3f24cf28362ab8f79b9e03 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T09:33:32.776 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 podman[87539]: 2024-09-17 09:33:32.395286113 +0000 UTC m=+2.858606201 container start 5f8842670a3c5e1e6e850c7abbe8958f2b6788df2b3f24cf28362ab8f79b9e03 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T09:33:32.776 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.398Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-17T09:33:32.776 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.399Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-17T09:33:32.776 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.399Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-17T09:33:32.776 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.399Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-17T09:33:32.776 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.399Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-17T09:33:32.776 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.399Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T09:33:32.776 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-17T09:33:32.776 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=arp 2024-09-17T09:33:32.776 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-17T09:33:32.776 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-17T09:33:32.777 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-17T09:33:32.777 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-17T09:33:32.777 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-17T09:33:32.777 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-17T09:33:32.777 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-17T09:33:32.777 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-17T09:33:32.777 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=edac 2024-09-17T09:33:32.778 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-17T09:33:32.778 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-17T09:33:32.778 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-17T09:33:32.778 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-17T09:33:32.778 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-17T09:33:32.778 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-17T09:33:32.778 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-17T09:33:32.778 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-17T09:33:32.778 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-17T09:33:32.778 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-17T09:33:32.778 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-17T09:33:32.778 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-17T09:33:32.778 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-17T09:33:32.779 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-17T09:33:32.779 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-17T09:33:32.779 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-17T09:33:32.779 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=os 2024-09-17T09:33:32.779 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-17T09:33:32.779 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-17T09:33:32.779 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-17T09:33:32.779 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-17T09:33:32.779 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-17T09:33:32.779 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-17T09:33:32.779 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-17T09:33:32.779 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=stat 2024-09-17T09:33:32.779 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-17T09:33:32.780 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-17T09:33:32.780 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-17T09:33:32.780 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=time 2024-09-17T09:33:32.780 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-17T09:33:32.780 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=uname 2024-09-17T09:33:32.780 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-17T09:33:32.780 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-17T09:33:32.780 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-17T09:33:32.780 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-17T09:33:32.780 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a[87684]: ts=2024-09-17T09:33:32.400Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-17T09:33:32.780 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 bash[87539]: 5f8842670a3c5e1e6e850c7abbe8958f2b6788df2b3f24cf28362ab8f79b9e03 2024-09-17T09:33:32.780 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:33:32 smithi029 systemd[1]: Started Ceph node-exporter.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:33:33.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[58326]: pgmap v248: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T09:33:33.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/536255971"}]': finished 2024-09-17T09:33:33.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[58326]: osdmap e137: 8 total, 8 up, 8 in 2024-09-17T09:33:33.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/1107772244' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3226471465"}]: dispatch 2024-09-17T09:33:33.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3226471465"}]: dispatch 2024-09-17T09:33:33.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:33.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:33.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[60489]: pgmap v248: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T09:33:33.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/536255971"}]': finished 2024-09-17T09:33:33.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[60489]: osdmap e137: 8 total, 8 up, 8 in 2024-09-17T09:33:33.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/1107772244' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3226471465"}]: dispatch 2024-09-17T09:33:33.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3226471465"}]: dispatch 2024-09-17T09:33:33.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:33.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:32 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:33.113 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:32 smithi103 ceph-mon[49967]: pgmap v248: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T09:33:33.113 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:32 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/536255971"}]': finished 2024-09-17T09:33:33.113 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:32 smithi103 ceph-mon[49967]: osdmap e137: 8 total, 8 up, 8 in 2024-09-17T09:33:33.113 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:32 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/1107772244' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3226471465"}]: dispatch 2024-09-17T09:33:33.113 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:32 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3226471465"}]: dispatch 2024-09-17T09:33:33.114 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:32 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:33.114 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:32 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:33 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3226471465"}]': finished 2024-09-17T09:33:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:33 smithi029 ceph-mon[58326]: osdmap e138: 8 total, 8 up, 8 in 2024-09-17T09:33:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:33 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/1538473274' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: dispatch 2024-09-17T09:33:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:33 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: dispatch 2024-09-17T09:33:34.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:33 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3226471465"}]': finished 2024-09-17T09:33:34.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:33 smithi029 ceph-mon[60489]: osdmap e138: 8 total, 8 up, 8 in 2024-09-17T09:33:34.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:33 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/1538473274' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: dispatch 2024-09-17T09:33:34.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:33 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: dispatch 2024-09-17T09:33:34.208 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:33 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/3226471465"}]': finished 2024-09-17T09:33:34.208 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:33 smithi103 ceph-mon[49967]: osdmap e138: 8 total, 8 up, 8 in 2024-09-17T09:33:34.208 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:33 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/1538473274' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: dispatch 2024-09-17T09:33:34.208 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:33 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]: dispatch 2024-09-17T09:33:34.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:34 smithi103 systemd[1]: Stopping Ceph node-exporter.b for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:33:34.889 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:34 smithi103 podman[68240]: 2024-09-17 09:33:34.573489332 +0000 UTC m=+0.128412667 container died 1adf1b5baad1b8aba6d426b76683866f49a51dbdb8219b0f7d6e10a61e6ffd4d (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T09:33:34.890 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:34 smithi103 ceph-mon[49967]: pgmap v251: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:34.890 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:34 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]': finished 2024-09-17T09:33:34.890 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:34 smithi103 ceph-mon[49967]: osdmap e139: 8 total, 8 up, 8 in 2024-09-17T09:33:34.890 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:34 smithi103 ceph-mon[49967]: Upgrade: Updating node-exporter.b (2/2) 2024-09-17T09:33:34.890 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:34 smithi103 ceph-mon[49967]: Deploying daemon node-exporter.b on smithi103 2024-09-17T09:33:34.890 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:34 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/2879707923' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: dispatch 2024-09-17T09:33:35.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:34 smithi029 ceph-mon[60489]: pgmap v251: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:35.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:34 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]': finished 2024-09-17T09:33:35.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:34 smithi029 ceph-mon[60489]: osdmap e139: 8 total, 8 up, 8 in 2024-09-17T09:33:35.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:34 smithi029 ceph-mon[60489]: Upgrade: Updating node-exporter.b (2/2) 2024-09-17T09:33:35.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:34 smithi029 ceph-mon[60489]: Deploying daemon node-exporter.b on smithi103 2024-09-17T09:33:35.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:34 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/2879707923' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: dispatch 2024-09-17T09:33:35.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:34 smithi029 ceph-mon[58326]: pgmap v251: 161 pgs: 161 active+clean; 604 KiB data, 320 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:35.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:34 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/2731392062"}]': finished 2024-09-17T09:33:35.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:34 smithi029 ceph-mon[58326]: osdmap e139: 8 total, 8 up, 8 in 2024-09-17T09:33:35.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:34 smithi029 ceph-mon[58326]: Upgrade: Updating node-exporter.b (2/2) 2024-09-17T09:33:35.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:34 smithi029 ceph-mon[58326]: Deploying daemon node-exporter.b on smithi103 2024-09-17T09:33:35.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:34 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/2879707923' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/225600821"}]: dispatch 2024-09-17T09:33:35.207 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:34 smithi103 podman[68240]: 2024-09-17 09:33:34.996506723 +0000 UTC m=+0.551430058 container cleanup 1adf1b5baad1b8aba6d426b76683866f49a51dbdb8219b0f7d6e10a61e6ffd4d (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T09:33:35.207 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:34 smithi103 bash[68240]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b 2024-09-17T09:33:35.460 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:35 smithi103 podman[68252]: 2024-09-17 09:33:35.206260901 +0000 UTC m=+0.636771403 container remove 1adf1b5baad1b8aba6d426b76683866f49a51dbdb8219b0f7d6e10a61e6ffd4d (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter.b, maintainer=The Prometheus Authors ) 2024-09-17T09:33:35.460 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:35 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:33:35.755 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:35 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-17T09:33:35.756 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:35 smithi103 systemd[1]: Stopped Ceph node-exporter.b for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:33:35.756 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:35 smithi103 systemd[1]: Starting Ceph node-exporter.b for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:33:36.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:35 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/2879707923' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/225600821"}]': finished 2024-09-17T09:33:36.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:35 smithi029 ceph-mon[60489]: osdmap e140: 8 total, 8 up, 8 in 2024-09-17T09:33:36.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:35 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/4013463370' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/4068330535"}]: dispatch 2024-09-17T09:33:36.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:35 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/2879707923' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/225600821"}]': finished 2024-09-17T09:33:36.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:35 smithi029 ceph-mon[58326]: osdmap e140: 8 total, 8 up, 8 in 2024-09-17T09:33:36.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:35 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/4013463370' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/4068330535"}]: dispatch 2024-09-17T09:33:36.188 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:35 smithi103 bash[68394]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-17T09:33:36.189 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:35 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/2879707923' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:0/225600821"}]': finished 2024-09-17T09:33:36.189 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:35 smithi103 ceph-mon[49967]: osdmap e140: 8 total, 8 up, 8 in 2024-09-17T09:33:36.189 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:35 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/4013463370' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/4068330535"}]: dispatch 2024-09-17T09:33:36.505 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:36 smithi103 bash[68394]: Getting image source signatures 2024-09-17T09:33:36.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:36 smithi103 bash[68394]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-17T09:33:36.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:36 smithi103 bash[68394]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-17T09:33:36.506 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:36 smithi103 bash[68394]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-17T09:33:37.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:36 smithi029 ceph-mon[60489]: pgmap v254: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:37.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:36 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/4013463370' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/4068330535"}]': finished 2024-09-17T09:33:37.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:36 smithi029 ceph-mon[60489]: osdmap e141: 8 total, 8 up, 8 in 2024-09-17T09:33:37.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:36 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/1495574500' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/395014989"}]: dispatch 2024-09-17T09:33:37.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:36 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/395014989"}]: dispatch 2024-09-17T09:33:37.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:36 smithi029 ceph-mon[58326]: pgmap v254: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:37.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:36 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/4013463370' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/4068330535"}]': finished 2024-09-17T09:33:37.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:36 smithi029 ceph-mon[58326]: osdmap e141: 8 total, 8 up, 8 in 2024-09-17T09:33:37.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:36 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/1495574500' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/395014989"}]: dispatch 2024-09-17T09:33:37.049 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:36 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/395014989"}]: dispatch 2024-09-17T09:33:37.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:36 smithi103 ceph-mon[49967]: pgmap v254: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:37.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:36 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/4013463370' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/4068330535"}]': finished 2024-09-17T09:33:37.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:36 smithi103 ceph-mon[49967]: osdmap e141: 8 total, 8 up, 8 in 2024-09-17T09:33:37.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:36 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/1495574500' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/395014989"}]: dispatch 2024-09-17T09:33:37.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:36 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/395014989"}]: dispatch 2024-09-17T09:33:37.839 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:37 smithi103 bash[68394]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-17T09:33:37.839 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:37 smithi103 bash[68394]: Writing manifest to image destination 2024-09-17T09:33:37.840 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:37 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/395014989"}]': finished 2024-09-17T09:33:37.840 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:37 smithi103 ceph-mon[49967]: osdmap e142: 8 total, 8 up, 8 in 2024-09-17T09:33:37.840 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:37 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/1016884013' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/395014989"}]: dispatch 2024-09-17T09:33:37.840 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:37 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/395014989"}]: dispatch 2024-09-17T09:33:38.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:37 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/395014989"}]': finished 2024-09-17T09:33:38.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:37 smithi029 ceph-mon[58326]: osdmap e142: 8 total, 8 up, 8 in 2024-09-17T09:33:38.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:37 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/1016884013' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/395014989"}]: dispatch 2024-09-17T09:33:38.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:37 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/395014989"}]: dispatch 2024-09-17T09:33:38.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:37 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6800/395014989"}]': finished 2024-09-17T09:33:38.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:37 smithi029 ceph-mon[60489]: osdmap e142: 8 total, 8 up, 8 in 2024-09-17T09:33:38.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:37 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/1016884013' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/395014989"}]: dispatch 2024-09-17T09:33:38.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:37 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/395014989"}]: dispatch 2024-09-17T09:33:38.422 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 podman[68394]: 2024-09-17 09:33:38.077995327 +0000 UTC m=+2.299505267 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-17T09:33:38.422 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 podman[68394]: 2024-09-17 09:33:38.167662689 +0000 UTC m=+2.389172627 container create 1f9c124dcc9ad4f920e6de8fbb3e6e52a4971296508f0e99141a776f70e15b07 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T09:33:38.757 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 podman[68394]: 2024-09-17 09:33:38.422024554 +0000 UTC m=+2.643534492 container init 1f9c124dcc9ad4f920e6de8fbb3e6e52a4971296508f0e99141a776f70e15b07 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T09:33:38.757 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 podman[68394]: 2024-09-17 09:33:38.425445655 +0000 UTC m=+2.646955598 container start 1f9c124dcc9ad4f920e6de8fbb3e6e52a4971296508f0e99141a776f70e15b07 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T09:33:38.757 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.429Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-17T09:33:38.757 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.429Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-17T09:33:38.757 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.430Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-17T09:33:38.757 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.430Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-17T09:33:38.758 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.430Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-17T09:33:38.758 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.430Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T09:33:38.758 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-17T09:33:38.758 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=arp 2024-09-17T09:33:38.758 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-17T09:33:38.758 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-17T09:33:38.758 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-17T09:33:38.758 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-17T09:33:38.758 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-17T09:33:38.758 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-17T09:33:38.758 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-17T09:33:38.758 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=edac 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-17T09:33:38.759 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-17T09:33:38.760 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-17T09:33:38.760 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-17T09:33:38.760 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-17T09:33:38.760 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=os 2024-09-17T09:33:38.760 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-17T09:33:38.760 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-17T09:33:38.760 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-17T09:33:38.760 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-17T09:33:38.760 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-17T09:33:38.760 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-17T09:33:38.761 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-17T09:33:38.761 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=stat 2024-09-17T09:33:38.761 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-17T09:33:38.761 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-17T09:33:38.761 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-17T09:33:38.761 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=time 2024-09-17T09:33:38.761 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-17T09:33:38.761 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=uname 2024-09-17T09:33:38.761 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-17T09:33:38.761 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-17T09:33:38.761 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-17T09:33:38.762 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.431Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-17T09:33:38.762 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b[68468]: ts=2024-09-17T09:33:38.432Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-17T09:33:38.762 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 bash[68394]: 1f9c124dcc9ad4f920e6de8fbb3e6e52a4971296508f0e99141a776f70e15b07 2024-09-17T09:33:38.762 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:33:38 smithi103 systemd[1]: Started Ceph node-exporter.b for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:33:39.131 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:38 smithi029 ceph-mon[60489]: pgmap v257: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:39.131 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:38 smithi029 ceph-mon[60489]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/395014989"}]': finished 2024-09-17T09:33:39.131 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:38 smithi029 ceph-mon[60489]: osdmap e143: 8 total, 8 up, 8 in 2024-09-17T09:33:39.131 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:38 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:33:39.131 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:39.131 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:38 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:39.131 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:38 smithi029 ceph-mon[58326]: pgmap v257: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:39.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:38 smithi029 ceph-mon[58326]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/395014989"}]': finished 2024-09-17T09:33:39.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:38 smithi029 ceph-mon[58326]: osdmap e143: 8 total, 8 up, 8 in 2024-09-17T09:33:39.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:38 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:33:39.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:39.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:38 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:39.151 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-mon[49967]: pgmap v257: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:39.151 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-mon[49967]: from='client.? ' entity='client.iscsi.foo.smithi029.gellkh' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.29:6801/395014989"}]': finished 2024-09-17T09:33:39.151 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-mon[49967]: osdmap e143: 8 total, 8 up, 8 in 2024-09-17T09:33:39.151 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:33:39.151 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:39.151 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:38 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:41.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:40 smithi103 ceph-mon[49967]: pgmap v259: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T09:33:41.140 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:40 smithi029 ceph-mon[60489]: pgmap v259: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T09:33:41.141 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:40 smithi029 ceph-mon[58326]: pgmap v259: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T09:33:43.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:42 smithi103 ceph-mon[49967]: pgmap v260: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:33:43.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:42 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:42 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:42 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:42 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:42 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:42 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:42 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:42 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.228 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[58326]: pgmap v260: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:33:43.228 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.228 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.228 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.229 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.229 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.229 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.229 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.229 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.229 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[60489]: pgmap v260: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:33:43.229 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.229 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.231 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.232 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.232 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.232 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.232 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:43.232 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:42 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:45.251 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:44 smithi103 ceph-mon[49967]: pgmap v261: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T09:33:45.288 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:44 smithi029 ceph-mon[58326]: pgmap v261: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T09:33:45.288 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:44 smithi029 ceph-mon[60489]: pgmap v261: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T09:33:46.847 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:46.847 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:46.847 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: pgmap v262: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T09:33:46.847 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:46.847 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:46.847 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:46.847 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:33:46.847 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:33:46.847 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: pgmap v263: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 610 B/s rd, 0 op/s 2024-09-17T09:33:46.847 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: pgmap v264: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-09-17T09:33:46.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:46.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:33:46.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:46.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:46.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:46.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:46.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:46.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:46.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:46.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:46.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:46.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:46.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:46.849 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:46.849 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:46 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:47.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: pgmap v262: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T09:33:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:33:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:33:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: pgmap v263: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 610 B/s rd, 0 op/s 2024-09-17T09:33:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: pgmap v264: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-09-17T09:33:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:33:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: pgmap v262: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: pgmap v263: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 610 B/s rd, 0 op/s 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: pgmap v264: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:47.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:46 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:33:49.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:49 smithi103 ceph-mon[49967]: Upgrade: Updating prometheus.a 2024-09-17T09:33:49.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:49 smithi103 ceph-mon[49967]: Deploying daemon prometheus.a on smithi103 2024-09-17T09:33:49.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:49 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:33:49.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:49 smithi103 ceph-mon[49967]: pgmap v265: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:49.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:49 smithi029 ceph-mon[58326]: Upgrade: Updating prometheus.a 2024-09-17T09:33:49.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:49 smithi029 ceph-mon[58326]: Deploying daemon prometheus.a on smithi103 2024-09-17T09:33:49.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:49 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:33:49.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:49 smithi029 ceph-mon[58326]: pgmap v265: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:49.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:49 smithi029 ceph-mon[60489]: Upgrade: Updating prometheus.a 2024-09-17T09:33:49.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:49 smithi029 ceph-mon[60489]: Deploying daemon prometheus.a on smithi103 2024-09-17T09:33:49.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:49 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:33:49.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:49 smithi029 ceph-mon[60489]: pgmap v265: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:51.421 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-mon[49967]: pgmap v266: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:51.421 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 systemd[1]: Starting Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:33:51.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:51 smithi029 ceph-mon[58326]: pgmap v266: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:51.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:51 smithi029 ceph-mon[60489]: pgmap v266: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:51.679 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 podman[72056]: 2024-09-17 09:33:51.32323046 +0000 UTC m=+0.017119061 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-17T09:33:51.679 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 podman[72056]: 2024-09-17 09:33:51.420567667 +0000 UTC m=+0.114456257 container create f332a13317762305bea7f335cdc9f6972bebee8c65f6e2778d183a4fa12a885f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 podman[72056]: 2024-09-17 09:33:51.614668969 +0000 UTC m=+0.308557570 container init f332a13317762305bea7f335cdc9f6972bebee8c65f6e2778d183a4fa12a885f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 podman[72056]: 2024-09-17 09:33:51.617195517 +0000 UTC m=+0.311084115 container start f332a13317762305bea7f335cdc9f6972bebee8c65f6e2778d183a4fa12a885f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.641Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.641Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.641Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi103 (none))" 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.641Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.641Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.642Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.643Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.643Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/prometheus/lock 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.645Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.645Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.646Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-17T09:33:51.680 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.646Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.537µs 2024-09-17T09:33:51.681 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.646Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-17T09:33:51.681 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.646Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=17 2024-09-17T09:33:51.681 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.653Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=17 2024-09-17T09:33:51.681 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.653Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=17 2024-09-17T09:33:51.681 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.653Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=17 2024-09-17T09:33:51.681 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.653Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=17 2024-09-17T09:33:51.681 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.653Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=17 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.654Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=17 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.654Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=7 maxSegment=17 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.654Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=8 maxSegment=17 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.654Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=9 maxSegment=17 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.654Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=10 maxSegment=17 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.654Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=11 maxSegment=17 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.655Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=12 maxSegment=17 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.656Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=13 maxSegment=17 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.657Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=14 maxSegment=17 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.657Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=15 maxSegment=17 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.657Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=16 maxSegment=17 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.657Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=17 maxSegment=17 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.657Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=35.681µs wal_replay_duration=11.311407ms wbl_replay_duration=117ns total_replay_duration=11.366196ms 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.659Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-17T09:33:51.682 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.659Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-17T09:33:51.683 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.659Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T09:33:51.683 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.678Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.498017ms db_storage=735ns remote_storage=1.16µs web_handler=378ns query_engine=888ns scrape=411.93µs scrape_sd=83.998µs notify=33.11µs notify_sd=24.863µs rules=18.625689ms tracing=6.25µs 2024-09-17T09:33:51.683 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.678Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-17T09:33:51.683 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:33:51.678Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-17T09:33:52.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 bash[72056]: f332a13317762305bea7f335cdc9f6972bebee8c65f6e2778d183a4fa12a885f 2024-09-17T09:33:52.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:33:51 smithi103 systemd[1]: Started Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:33:53.240 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:52 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:53.240 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:52 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:53.240 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:52 smithi103 ceph-mon[49967]: pgmap v267: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:53.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:52 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:53.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:52 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:53.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:52 smithi029 ceph-mon[58326]: pgmap v267: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:53.245 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:52 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:53.245 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:52 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:53.245 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:52 smithi029 ceph-mon[60489]: pgmap v267: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:55.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:55 smithi103 ceph-mon[49967]: pgmap v268: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:55.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:55 smithi029 ceph-mon[58326]: pgmap v268: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:55.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:55 smithi029 ceph-mon[60489]: pgmap v268: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:56.635 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:56 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.636 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:56 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.636 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:56 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.636 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:56 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.636 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:56 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.636 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:56 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.636 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:56 smithi103 ceph-mon[49967]: pgmap v269: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T09:33:56.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[60489]: pgmap v269: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T09:33:56.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.799 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:33:56.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:56 smithi029 ceph-mon[58326]: pgmap v269: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T09:33:59.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:59 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:33:59.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:33:59 smithi103 ceph-mon[49967]: pgmap v270: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:59.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:59 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:33:59.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:33:59 smithi029 ceph-mon[60489]: pgmap v270: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:33:59.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:59 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:33:59.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:33:59 smithi029 ceph-mon[58326]: pgmap v270: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:00.058 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:34:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: pgmap v271: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:34:00.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:00.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:00.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:00.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:00.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:00.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: pgmap v271: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:00.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:00.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:34:00.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:34:00.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:00 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: pgmap v271: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:34:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:34:00.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.507 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:00 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:00.733 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:34:00.733 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (7m) 5s ago 12m 19.1M - 0.20.0 0881eb8f169f 7bd158a6d65c 2024-09-17T09:34:00.733 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (7m) 5s ago 12m 36.9M - 6.7.4 557c83e11646 e361816f7c14 2024-09-17T09:34:00.733 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 running (43s) 5s ago 11m 50.5M - 3.8 980a0bfcd684 c5c886a0c37c 2024-09-17T09:34:00.733 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (6m) 5s ago 16m 439M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:34:00.733 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (7m) 5s ago 17m 495M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:34:00.734 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (6m) 5s ago 17m 61.8M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:34:00.734 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (5m) 5s ago 16m 46.5M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 da050285f124 2024-09-17T09:34:00.734 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (5m) 5s ago 16m 53.8M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 0d3d1a65e2c9 2024-09-17T09:34:00.734 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (28s) 5s ago 12m 3418k - 1.5.0 0da6a335fe13 5f8842670a3c 2024-09-17T09:34:00.734 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (22s) 5s ago 12m 3581k - 1.5.0 0da6a335fe13 1f9c124dcc9a 2024-09-17T09:34:00.734 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (4m) 5s ago 15m 64.9M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 4c456435000b 2024-09-17T09:34:00.734 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (4m) 5s ago 15m 58.7M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2ff250ddb1d9 2024-09-17T09:34:00.734 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (3m) 5s ago 15m 58.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7899f3c3c2f4 2024-09-17T09:34:00.734 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (3m) 5s ago 14m 63.3M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 94b84aca13ce 2024-09-17T09:34:00.734 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (3m) 5s ago 14m 56.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2306402255ea 2024-09-17T09:34:00.734 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (2m) 5s ago 14m 50.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0312d071d90b 2024-09-17T09:34:00.734 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (118s) 5s ago 13m 60.5M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 a5355c784b3c 2024-09-17T09:34:00.734 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (86s) 5s ago 13m 50.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 02f285335edf 2024-09-17T09:34:00.735 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 running (9s) 5s ago 13m 33.3M - 2.43.0 a07b618ecd1d f332a1331776 2024-09-17T09:34:00.735 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (71s) 5s ago 11m 77.2M - 18.2.4-855-g19e2a2be 980a0bfcd684 697c12a1dd89 2024-09-17T09:34:00.735 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (60s) 5s ago 11m 74.0M - 18.2.4-855-g19e2a2be 980a0bfcd684 6b0ca358485b 2024-09-17T09:34:00.735 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (65s) 5s ago 11m 75.2M - 18.2.4-855-g19e2a2be 980a0bfcd684 1b791ec24040 2024-09-17T09:34:00.735 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (55s) 5s ago 11m 75.1M - 18.2.4-855-g19e2a2be 980a0bfcd684 ddcc313b787c 2024-09-17T09:34:01.145 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:34:01.146 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:34:01.146 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:34:01.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:34:01.146 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:34:01.146 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:34:01.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:34:01.146 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:34:01.146 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T09:34:01.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:34:01.146 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:34:01.146 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T09:34:01.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:34:01.147 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:34:01.147 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 17 2024-09-17T09:34:01.147 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:34:01.147 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:34:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:01 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:34:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:01 smithi029 ceph-mon[60489]: from='client.55126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:01 smithi029 ceph-mon[60489]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T09:34:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:01 smithi029 ceph-mon[60489]: Cluster is now healthy 2024-09-17T09:34:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:01 smithi029 ceph-mon[60489]: from='client.55144 172.21.15.29:0/3046297985' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:01.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:01 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:34:01.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:01 smithi029 ceph-mon[58326]: from='client.55126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:01.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:01 smithi029 ceph-mon[58326]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T09:34:01.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:01 smithi029 ceph-mon[58326]: Cluster is now healthy 2024-09-17T09:34:01.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:01 smithi029 ceph-mon[58326]: from='client.55144 172.21.15.29:0/3046297985' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:01.492 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:34:01.492 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9", 2024-09-17T09:34:01.492 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": true, 2024-09-17T09:34:01.492 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T09:34:01.492 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [ 2024-09-17T09:34:01.492 INFO:teuthology.orchestra.run.smithi029.stdout: "osd", 2024-09-17T09:34:01.493 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr", 2024-09-17T09:34:01.493 INFO:teuthology.orchestra.run.smithi029.stdout: "iscsi", 2024-09-17T09:34:01.493 INFO:teuthology.orchestra.run.smithi029.stdout: "mon", 2024-09-17T09:34:01.493 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw" 2024-09-17T09:34:01.493 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-17T09:34:01.493 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": "18/23 daemons upgraded", 2024-09-17T09:34:01.493 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "Currently upgrading alertmanager daemons", 2024-09-17T09:34:01.493 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:34:01.493 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:34:01.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:01 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T09:34:01.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:01 smithi103 ceph-mon[49967]: from='client.55126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:01.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:01 smithi103 ceph-mon[49967]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T09:34:01.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:01 smithi103 ceph-mon[49967]: Cluster is now healthy 2024-09-17T09:34:01.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:01 smithi103 ceph-mon[49967]: from='client.55144 172.21.15.29:0/3046297985' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:01.897 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_OK 2024-09-17T09:34:02.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:02 smithi029 ceph-mon[58326]: from='client.55132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:02.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:02 smithi029 ceph-mon[58326]: from='client.45255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:02.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:02 smithi029 ceph-mon[58326]: Upgrade: Updating alertmanager.a 2024-09-17T09:34:02.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:02 smithi029 ceph-mon[58326]: Deploying daemon alertmanager.a on smithi029 2024-09-17T09:34:02.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:02 smithi029 ceph-mon[58326]: pgmap v272: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:02.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:02 smithi029 ceph-mon[58326]: from='client.45273 172.21.15.29:0/3143926917' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:34:02.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:02 smithi029 ceph-mon[60489]: from='client.55132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:02.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:02 smithi029 ceph-mon[60489]: from='client.45255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:02.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:02 smithi029 ceph-mon[60489]: Upgrade: Updating alertmanager.a 2024-09-17T09:34:02.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:02 smithi029 ceph-mon[60489]: Deploying daemon alertmanager.a on smithi029 2024-09-17T09:34:02.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:02 smithi029 ceph-mon[60489]: pgmap v272: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:02.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:02 smithi029 ceph-mon[60489]: from='client.45273 172.21.15.29:0/3143926917' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:34:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:02 smithi103 ceph-mon[49967]: from='client.55132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:02 smithi103 ceph-mon[49967]: from='client.45255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:02 smithi103 ceph-mon[49967]: Upgrade: Updating alertmanager.a 2024-09-17T09:34:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:02 smithi103 ceph-mon[49967]: Deploying daemon alertmanager.a on smithi029 2024-09-17T09:34:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:02 smithi103 ceph-mon[49967]: pgmap v272: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:02.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:02 smithi103 ceph-mon[49967]: from='client.45273 172.21.15.29:0/3143926917' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:34:03.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:03 smithi103 ceph-mon[49967]: from='client.55150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:03.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:03 smithi029 ceph-mon[58326]: from='client.55150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:03 smithi029 ceph-mon[60489]: from='client.55150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:04.506 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:04 smithi103 ceph-mon[49967]: pgmap v273: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:04.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:04 smithi029 ceph-mon[58326]: pgmap v273: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:04.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:04 smithi029 ceph-mon[60489]: pgmap v273: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:04.549 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:04 smithi029 systemd[1]: Stopping Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:34:04.549 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:04 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a[51806]: level=info ts=2024-09-17T09:34:04.290Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2024-09-17T09:34:04.549 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:04 smithi029 podman[92279]: 2024-09-17 09:34:04.311084458 +0000 UTC m=+0.176007996 container died 7bd158a6d65c037fc5b6057c7ae4511ae9d0a62ba41f3da07241c94162a0e93e (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:34:04.973 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:04 smithi029 podman[92279]: 2024-09-17 09:34:04.719496743 +0000 UTC m=+0.584420279 container cleanup 7bd158a6d65c037fc5b6057c7ae4511ae9d0a62ba41f3da07241c94162a0e93e (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:34:04.973 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:04 smithi029 bash[92279]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a 2024-09-17T09:34:05.298 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:04 smithi029 podman[92291]: 2024-09-17 09:34:04.972209074 +0000 UTC m=+0.655235422 container remove 7bd158a6d65c037fc5b6057c7ae4511ae9d0a62ba41f3da07241c94162a0e93e (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager.a, maintainer=The Prometheus Authors ) 2024-09-17T09:34:05.298 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:05 smithi029 podman[92291]: 2024-09-17 09:34:05.056504701 +0000 UTC m=+0.739531004 volume remove 40a502cae9ec4d68032d8de3f6a8d02cecd9e64afd5ecfec83ce3102a2814173 2024-09-17T09:34:05.737 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:05 smithi029 ceph-mon[58326]: pgmap v274: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:05.737 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:05 smithi029 ceph-mon[60489]: pgmap v274: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:05.738 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:05 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T09:34:05.738 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:05 smithi029 systemd[1]: Stopped Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:34:05.738 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:05 smithi029 systemd[1]: Starting Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:34:05.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:05 smithi103 ceph-mon[49967]: pgmap v274: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:06.048 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:05 smithi029 podman[92433]: 2024-09-17 09:34:05.639245117 +0000 UTC m=+0.016487550 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-17T09:34:06.048 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:05 smithi029 podman[92433]: 2024-09-17 09:34:05.743583071 +0000 UTC m=+0.120825487 volume create 0a11199cf45b9d2b2bf735f5a176fbd172612b5ad73abaa1d3f071e0d4a04e39 2024-09-17T09:34:06.048 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:05 smithi029 podman[92433]: 2024-09-17 09:34:05.861409213 +0000 UTC m=+0.238651628 container create e3a985039ca09301d543ecc67f157d13e82e1971150d5457008f00284a0f2600 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:34:06.509 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:06 smithi029 podman[92433]: 2024-09-17 09:34:06.183094458 +0000 UTC m=+0.560336884 container init e3a985039ca09301d543ecc67f157d13e82e1971150d5457008f00284a0f2600 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:34:06.509 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:06 smithi029 podman[92433]: 2024-09-17 09:34:06.18603348 +0000 UTC m=+0.563275896 container start e3a985039ca09301d543ecc67f157d13e82e1971150d5457008f00284a0f2600 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:34:06.509 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:34:06.198Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-17T09:34:06.510 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:34:06.198Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-17T09:34:06.510 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:34:06.200Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.29 port=9094 2024-09-17T09:34:06.510 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:34:06.201Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T09:34:06.510 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:34:06.225Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T09:34:06.510 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:34:06.225Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T09:34:06.510 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:34:06.228Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-17T09:34:06.510 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:06 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:34:06.228Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-17T09:34:06.510 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:06 smithi029 bash[92433]: e3a985039ca09301d543ecc67f157d13e82e1971150d5457008f00284a0f2600 2024-09-17T09:34:06.510 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:06 smithi029 systemd[1]: Started Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:34:07.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:07 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:07.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:07 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:07.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:07 smithi103 ceph-mon[49967]: pgmap v275: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-17T09:34:07.778 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:07 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:07.778 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:07 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:07.778 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:07 smithi029 ceph-mon[60489]: pgmap v275: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-17T09:34:07.778 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:07 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:07.778 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:07 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:07.778 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:07 smithi029 ceph-mon[58326]: pgmap v275: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-17T09:34:08.548 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:08 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:34:08.202Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000227294s 2024-09-17T09:34:08.960 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:08 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:08.960 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:08 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:09.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:08 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:10.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:09 smithi029 ceph-mon[60489]: pgmap v276: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:10.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:09 smithi029 ceph-mon[58326]: pgmap v276: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:10.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:09 smithi103 ceph-mon[49967]: pgmap v276: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-17T09:34:11.112 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:10 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:11.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:10 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:11.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:10 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:11.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:10 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:11.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:10 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:11.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:10 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:11.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:10 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:11.114 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:10 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:10 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:10 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:10 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:11.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:10 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:12.242 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:11 smithi029 ceph-mon[58326]: pgmap v277: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:12.243 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:11 smithi029 ceph-mon[60489]: pgmap v277: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:12.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:11 smithi103 ceph-mon[49967]: pgmap v277: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:13.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:13 smithi103 ceph-mon[49967]: pgmap v278: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:13.776 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:13 smithi029 ceph-mon[60489]: pgmap v278: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:13.776 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:13 smithi029 ceph-mon[58326]: pgmap v278: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:15.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:15.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:15.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:15.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.007 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.008 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:14 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:15.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:15.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:15.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:15.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:15.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:15.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:34:15.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:34:15.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:34:15.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:15.300 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:15.302 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:14 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:16.077 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:15 smithi103 ceph-mon[49967]: pgmap v279: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:16.206 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:15 smithi029 ceph-mon[58326]: pgmap v279: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:16.206 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:15 smithi029 ceph-mon[60489]: pgmap v279: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:16.548 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:34:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:34:16.204Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002108067s 2024-09-17T09:34:17.065 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:16 smithi103 ceph-mon[49967]: Upgrade: Updating grafana.a 2024-09-17T09:34:17.065 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:16 smithi103 ceph-mon[49967]: Deploying daemon grafana.a on smithi103 2024-09-17T09:34:17.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:16 smithi029 ceph-mon[60489]: Upgrade: Updating grafana.a 2024-09-17T09:34:17.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:16 smithi029 ceph-mon[60489]: Deploying daemon grafana.a on smithi103 2024-09-17T09:34:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:16 smithi029 ceph-mon[58326]: Upgrade: Updating grafana.a 2024-09-17T09:34:17.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:16 smithi029 ceph-mon[58326]: Deploying daemon grafana.a on smithi103 2024-09-17T09:34:17.996 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:17 smithi103 ceph-mon[49967]: pgmap v280: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:18.065 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:17 smithi029 ceph-mon[58326]: pgmap v280: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:18.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:17 smithi029 ceph-mon[60489]: pgmap v280: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:18.693 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 systemd[1]: Stopping Ceph grafana.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:34:18.693 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:34:18+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-09-17T09:34:18.693 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:34:18+0000 lvl=info msg="Stopped CleanUpService" logger=server reason="context canceled" 2024-09-17T09:34:18.693 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:34:18+0000 lvl=info msg="Stopped provisioningServiceImpl" logger=server reason="context canceled" 2024-09-17T09:34:18.693 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:34:18+0000 lvl=info msg="Stopped UserAuthTokenService" logger=server reason="context canceled" 2024-09-17T09:34:18.693 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:34:18+0000 lvl=info msg="Stopped RemoteCache" logger=server reason="context canceled" 2024-09-17T09:34:18.694 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:34:18+0000 lvl=info msg="Stopped AlertEngine" logger=server reason="context canceled" 2024-09-17T09:34:18.694 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:34:18+0000 lvl=info msg="Stopped PluginManager" logger=server reason="context canceled" 2024-09-17T09:34:18.694 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:34:18+0000 lvl=info msg="Stopped InternalMetricsService" logger=server reason="context canceled" 2024-09-17T09:34:18.694 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:34:18+0000 lvl=info msg="Stopped BackendPluginManager" logger=server reason="context canceled" 2024-09-17T09:34:18.694 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:34:18+0000 lvl=info msg="Stopped Stream Manager" 2024-09-17T09:34:18.694 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:34:18+0000 lvl=info msg="Stopped UsageStatsService" logger=server reason="context canceled" 2024-09-17T09:34:18.694 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a[44058]: t=2024-09-17T09:34:18+0000 lvl=info msg="Stopped NotificationService" logger=server reason="context canceled" 2024-09-17T09:34:18.694 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 podman[75939]: 2024-09-17 09:34:18.435647317 +0000 UTC m=+0.102365989 container died e361816f7c1489fb4a6d6bd4714b3bacde0cca491f2057463ea954ac85f4459e (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a, org.label-schema.vendor=CentOS, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.19.8, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, description=Ceph Grafana Container, org.label-schema.name=CentOS Base Image, maintainer=Paul Cuzner ) 2024-09-17T09:34:19.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:18 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:19.006 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 podman[75939]: 2024-09-17 09:34:18.808292983 +0000 UTC m=+0.475011653 container cleanup e361816f7c1489fb4a6d6bd4714b3bacde0cca491f2057463ea954ac85f4459e (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.buildah.version=1.19.8) 2024-09-17T09:34:19.006 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:18 smithi103 bash[75939]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a 2024-09-17T09:34:19.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:18 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:19.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:18 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:19.366 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:19 smithi103 podman[75951]: 2024-09-17 09:34:19.101595187 +0000 UTC m=+0.666471714 container remove e361816f7c1489fb4a6d6bd4714b3bacde0cca491f2057463ea954ac85f4459e (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana.a, org.label-schema.vendor=CentOS, maintainer=Paul Cuzner , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, io.buildah.version=1.19.8, org.label-schema.build-date=20201204, summary=Grafana Container configured for Ceph mgr/dashboard integration, org.label-schema.schema-version=1.0, description=Ceph Grafana Container) 2024-09-17T09:34:19.756 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:19 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-17T09:34:19.756 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:19 smithi103 systemd[1]: Stopped Ceph grafana.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:34:19.756 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:19 smithi103 systemd[1]: Starting Ceph grafana.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:34:20.114 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:19 smithi103 ceph-mon[49967]: pgmap v281: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:20.114 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:19 smithi103 podman[76092]: 2024-09-17 09:34:19.70224289 +0000 UTC m=+0.016177462 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-17T09:34:20.114 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:19 smithi103 podman[76092]: 2024-09-17 09:34:19.861390348 +0000 UTC m=+0.175324927 container create 0053793acd878aa815380e7a6168461856eb3bf46a698426a9e4816059dca462 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, architecture=x86_64, description=Ceph Grafana Container, com.redhat.component=ubi8-container, vcs-type=git, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, name=ubi8, vendor=Red Hat, Inc., version=8.5, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public) 2024-09-17T09:34:20.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:19 smithi029 ceph-mon[58326]: pgmap v281: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:20.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:19 smithi029 ceph-mon[60489]: pgmap v281: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:20.365 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 podman[76092]: 2024-09-17 09:34:20.113576183 +0000 UTC m=+0.427510764 container init 0053793acd878aa815380e7a6168461856eb3bf46a698426a9e4816059dca462 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a, vendor=Red Hat, Inc., maintainer=Paul Cuzner , release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, version=8.5, description=Ceph Grafana Container, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-09-17T09:34:20.365 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 podman[76092]: 2024-09-17 09:34:20.116722775 +0000 UTC m=+0.430657347 container start 0053793acd878aa815380e7a6168461856eb3bf46a698426a9e4816059dca462 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.buildah.version=1.32.0, io.openshift.expose-services=, vendor=Red Hat, Inc., architecture=x86_64, build-date=2022-04-29T11:48:11.055897, vcs-type=git, distribution-scope=public, maintainer=Paul Cuzner , description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-09-17T09:34:20.366 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 bash[76092]: 0053793acd878aa815380e7a6168461856eb3bf46a698426a9e4816059dca462 2024-09-17T09:34:20.366 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 systemd[1]: Started Ceph grafana.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:34:20.366 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.307222883Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-17T09:34:20.366 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.307406498Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-17T09:34:20.366 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.307442851Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-17T09:34:20.366 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.307452421Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-17T09:34:20.367 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.307458893Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-17T09:34:20.367 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.307464827Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-17T09:34:20.367 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.307470634Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-17T09:34:20.367 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.307477159Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-17T09:34:20.367 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.307483322Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-17T09:34:20.367 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.307488981Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-17T09:34:20.367 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.3074945Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-17T09:34:20.367 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.307499993Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-17T09:34:20.367 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.30750578Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-17T09:34:20.367 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=settings t=2024-09-17T09:34:20.307511658Z level=info msg="App mode production" 2024-09-17T09:34:20.367 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=sqlstore t=2024-09-17T09:34:20.307573697Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-17T09:34:20.367 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=sqlstore t=2024-09-17T09:34:20.307601306Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-17T09:34:20.368 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.308729768Z level=info msg="Starting DB migrations" 2024-09-17T09:34:20.368 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.308809319Z level=info msg="Executing migration" id="create migration_log table" 2024-09-17T09:34:20.368 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.309324864Z level=info msg="Executing migration" id="create user table" 2024-09-17T09:34:20.368 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.309773607Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-17T09:34:20.368 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.312431901Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-17T09:34:20.368 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.315128271Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-17T09:34:20.368 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.317737869Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-17T09:34:20.368 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.320541152Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-17T09:34:20.368 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.323406624Z level=info msg="Executing migration" id="create user table v2" 2024-09-17T09:34:20.368 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.323896412Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-17T09:34:20.368 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.326702299Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-17T09:34:20.368 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.327172057Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-17T09:34:20.368 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.329496272Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-17T09:34:20.369 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.329894987Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-17T09:34:20.369 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.332295001Z level=info msg="Executing migration" id="Update user table charset" 2024-09-17T09:34:20.369 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.332632477Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-17T09:34:20.369 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.334992616Z level=info msg="Executing migration" id="Add missing user data" 2024-09-17T09:34:20.369 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.337687684Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-17T09:34:20.369 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.34030185Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-17T09:34:20.369 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.3429112Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-17T09:34:20.369 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.345530449Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-17T09:34:20.369 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.348219372Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-17T09:34:20.369 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.350829633Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-17T09:34:20.369 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.353551094Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-17T09:34:20.369 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.35610748Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-17T09:34:20.369 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.358848746Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-17T09:34:20.370 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.359259931Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-17T09:34:20.370 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.361984987Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-17T09:34:20.370 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.364632987Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-17T09:34:20.370 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.365069408Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-17T09:34:20.616 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.367688285Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-17T09:34:20.617 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.368107348Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-17T09:34:20.617 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.370460039Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-17T09:34:20.617 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.371270019Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-17T09:34:20.617 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.37174961Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-17T09:34:20.617 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.372498932Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-17T09:34:20.617 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.373271359Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-17T09:34:20.617 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.374156782Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-17T09:34:20.617 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.374507047Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-17T09:34:20.617 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.375288174Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-17T09:34:20.617 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.37610353Z level=info msg="Executing migration" id="create star table" 2024-09-17T09:34:20.617 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.376906593Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-17T09:34:20.618 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.377780546Z level=info msg="Executing migration" id="create org table v1" 2024-09-17T09:34:20.618 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.378569976Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-17T09:34:20.618 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.379416539Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-17T09:34:20.618 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.380204382Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-17T09:34:20.618 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.381045436Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-17T09:34:20.618 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.381552792Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-17T09:34:20.618 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.3823916Z level=info msg="Executing migration" id="Update org table charset" 2024-09-17T09:34:20.619 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.383193619Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-17T09:34:20.619 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.384056244Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-17T09:34:20.619 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.384373734Z level=info msg="Executing migration" id="create dashboard table" 2024-09-17T09:34:20.619 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.384947407Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-17T09:34:20.619 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.385764632Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-17T09:34:20.619 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.386192145Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-17T09:34:20.619 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.387045525Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-17T09:34:20.619 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.387885019Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-17T09:34:20.619 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.388710257Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-17T09:34:20.619 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.38955589Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-17T09:34:20.619 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.390387826Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-17T09:34:20.620 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.390814727Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-17T09:34:20.620 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.391273758Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-17T09:34:20.620 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.391646465Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-17T09:34:20.620 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.392062888Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-17T09:34:20.620 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.392350848Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-17T09:34:20.620 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.392869683Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-17T09:34:20.620 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.39334045Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-17T09:34:20.620 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.39387648Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-17T09:34:20.620 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.394279288Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-17T09:34:20.620 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.394791176Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-17T09:34:20.620 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.395233431Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-17T09:34:20.620 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.395666807Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-17T09:34:20.621 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.395974091Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-17T09:34:20.621 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.396259752Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-17T09:34:20.621 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.396818009Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-17T09:34:20.621 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.397312717Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-17T09:34:20.621 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.400053027Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-17T09:34:20.621 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.402678333Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-17T09:34:20.621 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.403025994Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-17T09:34:20.621 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.405761944Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-17T09:34:20.621 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.408427897Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-17T09:34:20.621 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.409269019Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-17T09:34:20.621 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.410132756Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-17T09:34:20.621 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.410996475Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-17T09:34:20.621 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.41190756Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.412312471Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.413162931Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.414005312Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.414330989Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.41519025Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.415981019Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.416834222Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.417682938Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.418533269Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.41940769Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.420315338Z level=info msg="Executing migration" id="create data_source table" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.421102709Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.421902036Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-17T09:34:20.622 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.422321591Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-17T09:34:20.623 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.423191686Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-17T09:34:20.623 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.424049438Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-17T09:34:20.623 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.425065219Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-17T09:34:20.623 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.425923923Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-17T09:34:20.623 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.426776029Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-17T09:34:20.623 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.427648695Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-17T09:34:20.623 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.428517242Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-17T09:34:20.624 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.429389651Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-17T09:34:20.624 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.430238943Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-17T09:34:20.625 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.431049509Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-17T09:34:20.625 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.431384222Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-17T09:34:20.625 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.431959262Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-17T09:34:20.625 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.432788631Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-17T09:34:20.625 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.433107971Z level=info msg="Executing migration" id="Add uid column" 2024-09-17T09:34:20.625 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.433693462Z level=info msg="Executing migration" id="Update uid value" 2024-09-17T09:34:20.625 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.434546762Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-17T09:34:20.625 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.434987624Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-17T09:34:20.625 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.435904928Z level=info msg="Executing migration" id="create api_key table" 2024-09-17T09:34:20.625 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.43872563Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-17T09:34:20.625 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.441567306Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-17T09:34:20.625 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.444453371Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-17T09:34:20.625 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.447405226Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-17T09:34:20.626 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.447872748Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-17T09:34:20.626 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.450667981Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-17T09:34:20.626 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.451116522Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-17T09:34:20.626 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.453559188Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-17T09:34:20.626 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.453961841Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-17T09:34:20.626 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.454347193Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-17T09:34:20.626 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.454794825Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-17T09:34:20.626 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.455230475Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-17T09:34:20.626 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.455572909Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-17T09:34:20.626 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.45594221Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-17T09:34:20.626 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.456211944Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-17T09:34:20.626 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.456779279Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-17T09:34:20.626 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.457328656Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-17T09:34:20.627 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.457658002Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-17T09:34:20.627 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.458223739Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-17T09:34:20.627 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.458808502Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-17T09:34:20.627 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.459226204Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-17T09:34:20.627 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.459625223Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-17T09:34:20.627 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.460034534Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-17T09:34:20.627 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.46049904Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-17T09:34:20.627 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.4609443Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-17T09:34:20.627 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.46137227Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-17T09:34:20.627 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.461706144Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-17T09:34:20.627 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.461980964Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-17T09:34:20.627 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.462550473Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-17T09:34:20.627 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.463129642Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-17T09:34:20.628 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.463437266Z level=info msg="Executing migration" id="create quota table v1" 2024-09-17T09:34:20.628 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.463856446Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-17T09:34:20.628 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.464485538Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-17T09:34:20.628 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.464813479Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-17T09:34:20.628 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.465224673Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-17T09:34:20.628 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.465681881Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-17T09:34:20.628 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.466270632Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-17T09:34:20.628 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.46658935Z level=info msg="Executing migration" id="create session table" 2024-09-17T09:34:20.628 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.467033789Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-17T09:34:20.628 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.467317839Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-17T09:34:20.628 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.467641984Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-17T09:34:20.628 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.468054843Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.46846574Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.46877171Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.469052201Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.469343203Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.469661258Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.470051215Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.470320988Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.471070605Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.471395989Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.471993945Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.472688587Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.472990846Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.473635978Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-17T09:34:20.629 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.474193967Z level=info msg="Executing migration" id="create alert table v1" 2024-09-17T09:34:20.630 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.47469422Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-17T09:34:20.630 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.475168192Z level=info msg="Executing migration" id="add index alert state" 2024-09-17T09:34:20.630 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.477496449Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-17T09:34:20.630 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.477968869Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-17T09:34:20.630 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.478379901Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-17T09:34:20.630 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.478869654Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-17T09:34:20.630 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.479465119Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-17T09:34:20.630 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.481836378Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-17T09:34:20.630 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.482362839Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-17T09:34:20.630 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.482975123Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-17T09:34:20.630 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.483410804Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-17T09:34:20.630 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.483918028Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-17T09:34:20.630 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.484303692Z level=info msg="Executing migration" id="Add column is_default" 2024-09-17T09:34:20.631 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.48496353Z level=info msg="Executing migration" id="Add column frequency" 2024-09-17T09:34:20.631 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.48559916Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-17T09:34:20.631 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.486461549Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-17T09:34:20.631 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.487335738Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-17T09:34:20.631 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.487780656Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-17T09:34:20.631 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.48860343Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-17T09:34:20.631 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.489530831Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-17T09:34:20.631 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.490345151Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-17T09:34:20.631 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.491201355Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-17T09:34:20.631 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.492046898Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-17T09:34:20.631 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.492895096Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-17T09:34:20.631 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.493755804Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-17T09:34:20.631 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.49457704Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.495431345Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.496288001Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.497142681Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.498027195Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.498875072Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.499719674Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.500589735Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.501443154Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.50233553Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.505210879Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.508009952Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.511200078Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.513385716Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-17T09:34:20.632 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.516144652Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-17T09:34:20.633 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.519037692Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-17T09:34:20.633 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.521630922Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-17T09:34:20.633 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.524374204Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-17T09:34:20.633 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.527006026Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-17T09:34:20.633 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.529805945Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-17T09:34:20.633 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.532622255Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-17T09:34:20.633 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.535379874Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-17T09:34:20.633 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.535914165Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-17T09:34:20.633 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.538752009Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-17T09:34:20.633 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.541550157Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-17T09:34:20.633 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.544319419Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-17T09:34:20.633 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.547085806Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-17T09:34:20.633 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.549927082Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-17T09:34:20.634 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.552496586Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-17T09:34:20.634 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.55360412Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-17T09:34:20.634 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.556278879Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-17T09:34:20.634 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.55899928Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-17T09:34:20.634 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.561684622Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-17T09:34:20.634 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.562069749Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-17T09:34:20.634 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.564656773Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-17T09:34:20.634 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.567428396Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-17T09:34:20.634 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.569980586Z level=info msg="Executing migration" id="Move region to single row" 2024-09-17T09:34:20.634 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.572629284Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-17T09:34:20.634 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.575279151Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-17T09:34:20.634 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.578099386Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-17T09:34:20.634 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.578692973Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-17T09:34:20.635 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.581300286Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-17T09:34:20.635 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.583940921Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-17T09:34:20.635 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.586592304Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-17T09:34:20.635 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.586953484Z level=info msg="Executing migration" id="create test_data table" 2024-09-17T09:34:20.635 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.587815231Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-17T09:34:20.635 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.58865306Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-17T09:34:20.635 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.589466171Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-17T09:34:20.635 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.590229446Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-17T09:34:20.635 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.591039119Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-17T09:34:20.635 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.591425501Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-17T09:34:20.635 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.592218481Z level=info msg="Executing migration" id="create team table" 2024-09-17T09:34:20.635 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.593053472Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-17T09:34:20.635 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.593836989Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-17T09:34:20.636 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.594680721Z level=info msg="Executing migration" id="create team member table" 2024-09-17T09:34:20.636 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.595486268Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-17T09:34:20.636 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.596283023Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-17T09:34:20.636 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.597076777Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-17T09:34:20.636 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.597889332Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-17T09:34:20.636 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.598647059Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-17T09:34:20.636 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.599440283Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-17T09:34:20.636 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.600290288Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-17T09:34:20.636 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.601105498Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-17T09:34:20.636 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.601976289Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-17T09:34:20.636 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.602819302Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-17T09:34:20.636 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.603663074Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-17T09:34:20.636 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.60455135Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.605368975Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.605842368Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.606689869Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.607457231Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.608541288Z level=info msg="Executing migration" id="create tag table" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.609173434Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.610040294Z level=info msg="Executing migration" id="create login attempt table" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.610626592Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.611488354Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.612295822Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.615323546Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.616170558Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.617074583Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-17T09:34:20.637 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.619822267Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-17T09:34:20.638 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.622717698Z level=info msg="Executing migration" id="create user auth table" 2024-09-17T09:34:20.867 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.62540038Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-17T09:34:20.867 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.628180518Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-17T09:34:20.868 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.628494458Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-17T09:34:20.868 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.631455194Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-17T09:34:20.868 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.634248754Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-17T09:34:20.868 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.636972523Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-17T09:34:20.868 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.639612493Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-17T09:34:20.868 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.642389727Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-17T09:34:20.868 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.645064644Z level=info msg="Executing migration" id="create server_lock table" 2024-09-17T09:34:20.868 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.647932835Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-17T09:34:20.868 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.650615138Z level=info msg="Executing migration" id="create user auth token table" 2024-09-17T09:34:20.868 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.65346975Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-17T09:34:20.868 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.653951145Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-17T09:34:20.868 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.654399122Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-17T09:34:20.869 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.654879305Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-17T09:34:20.869 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.655681251Z level=info msg="Executing migration" id="create cache_data table" 2024-09-17T09:34:20.869 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.657252276Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-17T09:34:20.869 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.658052999Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-17T09:34:20.869 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.658557447Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-17T09:34:20.869 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.659008194Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-17T09:34:20.869 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.659277515Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-17T09:34:20.869 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.65959338Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-17T09:34:20.869 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.66129216Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-17T09:34:20.869 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.661934649Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-17T09:34:20.869 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.662371627Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-17T09:34:20.869 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.662686125Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-17T09:34:20.870 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.663189977Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-17T09:34:20.870 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.663610632Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-17T09:34:20.870 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.664027196Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-17T09:34:20.870 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.664449669Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-17T09:34:20.870 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.665231098Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-17T09:34:20.870 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.665671227Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-17T09:34:20.870 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.665977502Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-17T09:34:20.870 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.666360946Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-17T09:34:20.870 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.666791684Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-17T09:34:20.870 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.667202226Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-17T09:34:20.870 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.667470276Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-17T09:34:20.872 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.667899383Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-17T09:34:20.872 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.668273743Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-17T09:34:20.872 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.668698639Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-17T09:34:20.872 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.669121034Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-17T09:34:20.872 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.669947683Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-17T09:34:20.872 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.670359594Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-17T09:34:20.872 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.670779081Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-17T09:34:20.872 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.67419147Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-17T09:34:20.872 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.677519974Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-17T09:34:20.873 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.677953993Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-17T09:34:20.873 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.678343277Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-17T09:34:20.873 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.679197122Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-17T09:34:20.873 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.679666351Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-17T09:34:20.873 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.680168479Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-17T09:34:20.873 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.68303104Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-17T09:34:20.873 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.685785758Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-17T09:34:20.873 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.688535795Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-17T09:34:20.873 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.691391796Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-17T09:34:20.873 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.694202841Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-17T09:34:20.873 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.695213967Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-17T09:34:20.873 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.697607334Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-17T09:34:20.874 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.700248314Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-17T09:34:20.874 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.703166492Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-17T09:34:20.874 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.70420249Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-17T09:34:20.874 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.706705187Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-17T09:34:20.874 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.709438905Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-17T09:34:20.874 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.711948915Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-17T09:34:20.874 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.714376989Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-17T09:34:20.874 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.717092031Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-17T09:34:20.874 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.719547911Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-17T09:34:20.874 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.720089026Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-17T09:34:20.874 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.720399701Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-17T09:34:20.874 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.721299806Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.723180884Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.724081011Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.726971365Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.729556726Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.732327293Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.733186015Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.7340638Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.734958037Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.735903178Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.736686221Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.737559769Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.738451969Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.739276284Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-17T09:34:20.875 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.740155586Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-17T09:34:20.876 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.741035322Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-17T09:34:20.876 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.741882566Z level=info msg="Executing migration" id="create alert_image table" 2024-09-17T09:34:20.876 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.742678229Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-17T09:34:20.876 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.743542209Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-17T09:34:20.876 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.744396445Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-17T09:34:20.876 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.745109416Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-17T09:34:20.876 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.745574243Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-17T09:34:20.876 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.745676325Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-17T09:34:20.876 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.746545289Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-17T09:34:20.876 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.747280111Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-17T09:34:20.876 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.748203155Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-17T09:34:20.876 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.748480473Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-17T09:34:20.877 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.748619253Z level=info msg="alerts found to migrate" alerts=0 2024-09-17T09:34:20.877 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.749092044Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-17T09:34:20.877 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.749943981Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-17T09:34:20.877 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.750932551Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-17T09:34:20.877 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.751620155Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-17T09:34:20.877 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.752643143Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-17T09:34:20.877 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.753301719Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-17T09:34:20.877 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.753644845Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-17T09:34:20.877 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.754544012Z level=info msg="Executing migration" id="create data_keys table" 2024-09-17T09:34:20.877 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.755551265Z level=info msg="Executing migration" id="create secrets table" 2024-09-17T09:34:20.877 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.756397484Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-17T09:34:20.877 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.76091097Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-17T09:34:20.877 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.762026139Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-17T09:34:20.878 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.762927564Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-17T09:34:20.878 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.767138351Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-17T09:34:20.878 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.771610894Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-17T09:34:20.878 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.772502955Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-17T09:34:20.878 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.773343836Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-17T09:34:20.878 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.776060989Z level=info msg="Executing migration" id="create permission table" 2024-09-17T09:34:20.878 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.778902051Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-17T09:34:20.878 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.781905709Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-17T09:34:20.878 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.784893634Z level=info msg="Executing migration" id="create role table" 2024-09-17T09:34:20.878 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.788193127Z level=info msg="Executing migration" id="add column display_name" 2024-09-17T09:34:20.878 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.789154378Z level=info msg="Executing migration" id="add column group_name" 2024-09-17T09:34:20.878 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.792066795Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-17T09:34:20.879 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.794894117Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-17T09:34:20.879 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.797806301Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-17T09:34:20.879 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.800442672Z level=info msg="Executing migration" id="create team role table" 2024-09-17T09:34:20.879 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.803275103Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-17T09:34:20.879 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.803717573Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-17T09:34:20.879 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.804578042Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-17T09:34:20.879 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.805433812Z level=info msg="Executing migration" id="create user role table" 2024-09-17T09:34:20.879 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.806269756Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-17T09:34:20.879 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.80712167Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-17T09:34:20.879 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.807990422Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-17T09:34:20.879 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.80883837Z level=info msg="Executing migration" id="create builtin role table" 2024-09-17T09:34:20.879 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.809618254Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-17T09:34:20.879 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.810057772Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.810928716Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.811900712Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.812750997Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.813608528Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.814597609Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.815296549Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.816875063Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.818970478Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.821881557Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.824568598Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.827354869Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.830077607Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.833074921Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-17T09:34:20.880 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.835652683Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-17T09:34:20.881 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.83603237Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-17T09:34:20.881 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.838704545Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-17T09:34:20.881 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.839313979Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-17T09:34:20.881 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.841670551Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-17T09:34:20.881 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.844253655Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-17T09:34:20.881 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.846913339Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-17T09:34:20.881 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.849697468Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-17T09:34:20.881 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.852468373Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-17T09:34:20.881 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.855275676Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-17T09:34:20.881 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.858117648Z level=info msg="Executing migration" id="add correlation config column" 2024-09-17T09:34:20.881 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.860816062Z level=info msg="Executing migration" id="create entity_events table" 2024-09-17T09:34:20.881 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.863443071Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-17T09:34:20.882 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.866057953Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-17T09:34:20.882 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.866322201Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-17T09:34:20.882 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.868581182Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T09:34:20.882 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.868701295Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T09:34:21.171 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.871298216Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-17T09:34:21.171 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.873899162Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-17T09:34:21.172 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.876640359Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-17T09:34:21.172 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.877095744Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-17T09:34:21.172 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.879805205Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-17T09:34:21.172 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.882465336Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-17T09:34:21.172 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.882930999Z level=info msg="Executing migration" id="Drop public config table" 2024-09-17T09:34:21.172 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.885585907Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-17T09:34:21.172 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.886026012Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-17T09:34:21.172 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.888478034Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-17T09:34:21.172 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.889066563Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-17T09:34:21.172 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.891001536Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-17T09:34:21.172 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.894894295Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-17T09:34:21.172 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.897758612Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-17T09:34:21.173 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.898789961Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-17T09:34:21.173 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.899732854Z level=info msg="Executing migration" id="add share column" 2024-09-17T09:34:21.173 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.900900168Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-17T09:34:21.173 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.901809856Z level=info msg="Executing migration" id="create file table" 2024-09-17T09:34:21.173 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.902523296Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-17T09:34:21.173 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.903494988Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-17T09:34:21.173 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.904154855Z level=info msg="Executing migration" id="create file_meta table" 2024-09-17T09:34:21.173 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.905073322Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-17T09:34:21.173 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.905790404Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-17T09:34:21.173 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.906616978Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-17T09:34:21.173 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.907464708Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-17T09:34:21.173 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.907795278Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-17T09:34:21.173 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.908578557Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-17T09:34:21.174 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.909603607Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-17T09:34:21.174 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.909928232Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-17T09:34:21.174 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.910704358Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-17T09:34:21.174 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.911119344Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-17T09:34:21.174 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.912005868Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-17T09:34:21.174 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.912614733Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-17T09:34:21.174 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.913629342Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-17T09:34:21.174 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.914687491Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-17T09:34:21.174 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.915566473Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-17T09:34:21.174 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.926270058Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-17T09:34:21.175 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.927104011Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-17T09:34:21.175 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.927950641Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-17T09:34:21.175 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.931451611Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-17T09:34:21.175 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.932237649Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-17T09:34:21.175 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.93312606Z level=info msg="Executing migration" id="create folder table" 2024-09-17T09:34:21.175 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.93397946Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-17T09:34:21.175 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.934858749Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-17T09:34:21.175 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.935699803Z level=info msg="Executing migration" id="Update folder title length" 2024-09-17T09:34:21.175 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.935995192Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-17T09:34:21.175 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=migrator t=2024-09-17T09:34:20.93689853Z level=info msg="migrations completed" performed=485 skipped=0 duration=628.10923ms 2024-09-17T09:34:21.175 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=sqlstore t=2024-09-17T09:34:20.937535654Z level=info msg="Created default organization" 2024-09-17T09:34:21.175 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:20 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=plugin.loader t=2024-09-17T09:34:20.997750443Z level=info msg="Plugin registered" pluginID=input 2024-09-17T09:34:21.176 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=plugin.loader t=2024-09-17T09:34:21.025811958Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-17T09:34:21.176 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=plugin.loader t=2024-09-17T09:34:21.025845443Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-17T09:34:21.176 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=secrets t=2024-09-17T09:34:21.025917659Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-17T09:34:21.176 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=query_data t=2024-09-17T09:34:21.027668629Z level=info msg="Query Service initialization" 2024-09-17T09:34:21.176 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=live.push_http t=2024-09-17T09:34:21.030825561Z level=info msg="Live Push Gateway initialization" 2024-09-17T09:34:21.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:21.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:21.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:21 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:21.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:21 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:21.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:21 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:21.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:21 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:22.108 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=infra.usagestats.collector t=2024-09-17T09:34:21.818194726Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-17T09:34:22.108 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=provisioning.datasources t=2024-09-17T09:34:21.874429198Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-17T09:34:22.109 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=provisioning.datasources t=2024-09-17T09:34:21.907214794Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-17T09:34:22.109 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=provisioning.alerting t=2024-09-17T09:34:21.928741646Z level=info msg="starting to provision alerting" 2024-09-17T09:34:22.109 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=provisioning.alerting t=2024-09-17T09:34:21.928766596Z level=info msg="finished to provision alerting" 2024-09-17T09:34:22.109 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=grafanaStorageLogger t=2024-09-17T09:34:21.928917984Z level=info msg="storage starting" 2024-09-17T09:34:22.109 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=ngalert.state.manager t=2024-09-17T09:34:21.928943433Z level=info msg="Warming state cache for startup" 2024-09-17T09:34:22.109 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=http.server t=2024-09-17T09:34:21.930803957Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-17T09:34:22.109 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=ngalert.state.manager t=2024-09-17T09:34:21.939171395Z level=info msg="State cache has been initialized" states=0 duration=10.226776ms 2024-09-17T09:34:22.109 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=sqlstore.transactions t=2024-09-17T09:34:21.939208077Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T09:34:22.109 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=ngalert.multiorg.alertmanager t=2024-09-17T09:34:21.939229545Z level=info msg="starting MultiOrg Alertmanager" 2024-09-17T09:34:22.109 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=ticker t=2024-09-17T09:34:21.93927049Z level=info msg=starting first_tick=2024-09-17T09:34:30Z 2024-09-17T09:34:22.109 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:21 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=sqlstore.transactions t=2024-09-17T09:34:21.992232604Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T09:34:22.109 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=sqlstore.transactions t=2024-09-17T09:34:22.002522858Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-17T09:34:22.110 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=sqlstore.transactions t=2024-09-17T09:34:22.014816176Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T09:34:22.110 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:34:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a[76114]: logger=sqlstore.transactions t=2024-09-17T09:34:22.04019495Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-17T09:34:22.505 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:22 smithi103 ceph-mon[49967]: pgmap v282: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:22.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:22 smithi029 ceph-mon[58326]: pgmap v282: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:22.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:22 smithi029 ceph-mon[60489]: pgmap v282: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:24.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:23 smithi029 ceph-mon[58326]: pgmap v283: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:24.054 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:23 smithi029 ceph-mon[60489]: pgmap v283: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:24.255 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:23 smithi103 ceph-mon[49967]: pgmap v283: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:25.004 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:24 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:25.004 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:24 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:25.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:24 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:25.005 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:24 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:25.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:24 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:25.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:24 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:25.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:24 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:25.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:24 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:25.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:24 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:25.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:24 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:25.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:24 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:25.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:24 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:26.049 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:25 smithi103 ceph-mon[49967]: pgmap v284: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:26.199 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:25 smithi029 ceph-mon[60489]: pgmap v284: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:26.200 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:25 smithi029 ceph-mon[58326]: pgmap v284: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:27.450 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:27 smithi103 ceph-mon[49967]: pgmap v285: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:27.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:27 smithi029 ceph-mon[58326]: pgmap v285: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:27.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:27 smithi029 ceph-mon[60489]: pgmap v285: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:28.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:28.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:28.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:28.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:34:28.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:34:28.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:34:28.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.757 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.758 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: Upgrade: Finalizing container_image settings 2024-09-17T09:34:28.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-17T09:34:28.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-17T09:34:28.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-17T09:34:28.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-17T09:34:28.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-17T09:34:28.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-17T09:34:28.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-09-17T09:34:28.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-17T09:34:28.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-09-17T09:34:28.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-17T09:34:28.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-17T09:34:28.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-17T09:34:28.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-17T09:34:28.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-17T09:34:28.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-17T09:34:28.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-17T09:34:28.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-17T09:34:28.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-17T09:34:28.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-17T09:34:28.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-17T09:34:28.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-17T09:34:28.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-09-17T09:34:28.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-09-17T09:34:28.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: Upgrade: Complete! 2024-09-17T09:34:28.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-17T09:34:28.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-17T09:34:28.761 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:28.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:28.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:28.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:28.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:28.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:28.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.762 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:28 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:28.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:28.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:28.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:28.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: Upgrade: Finalizing container_image settings 2024-09-17T09:34:28.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:28.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: Upgrade: Finalizing container_image settings 2024-09-17T09:34:28.803 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-17T09:34:28.804 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-17T09:34:28.805 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.805 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-17T09:34:28.805 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-17T09:34:28.805 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-17T09:34:28.805 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-17T09:34:28.805 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-17T09:34:28.805 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-17T09:34:28.805 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-17T09:34:28.805 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-17T09:34:28.805 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-17T09:34:28.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-17T09:34:28.807 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-17T09:34:28.807 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-17T09:34:28.807 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-17T09:34:28.807 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-17T09:34:28.807 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-17T09:34:28.807 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-17T09:34:28.807 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.807 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.807 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.807 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.807 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.807 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.808 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-09-17T09:34:28.808 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-09-17T09:34:28.808 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: Upgrade: Complete! 2024-09-17T09:34:28.808 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-17T09:34:28.808 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-17T09:34:28.808 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:28.808 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:28.808 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:28.808 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.809 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:28.809 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:28.809 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:28.809 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.809 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:28.809 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-17T09:34:28.809 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.809 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.809 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.809 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.809 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: Upgrade: Complete! 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:28.810 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:28 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:29.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:29 smithi103 ceph-mon[49967]: pgmap v286: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:29.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:29 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:29.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:29 smithi029 ceph-mon[58326]: pgmap v286: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:29.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:29 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:29.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:29 smithi029 ceph-mon[60489]: pgmap v286: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:29.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:29 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:34:31.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:31 smithi103 ceph-mon[49967]: pgmap v287: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:31.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:31 smithi029 ceph-mon[58326]: pgmap v287: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:31.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:31 smithi029 ceph-mon[60489]: pgmap v287: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:32.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:32 smithi029 ceph-mon[58326]: from='client.55162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:32.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:32 smithi029 ceph-mon[60489]: from='client.55162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:32.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:32 smithi103 ceph-mon[49967]: from='client.55162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:32.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch ps' 2024-09-17T09:34:33.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:33 smithi029 ceph-mon[60489]: pgmap v288: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:33.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:33 smithi029 ceph-mon[58326]: pgmap v288: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:33.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:33 smithi103 ceph-mon[49967]: pgmap v288: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:35.088 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (29s) 25s ago 12m 16.4M - 0.25.0 c8568f914cd2 e3a985039ca0 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (15s) 11s ago 12m 27.4M - 9.4.7 954c08fa6188 0053793acd87 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 running (77s) 25s ago 11m 50.6M - 3.8 980a0bfcd684 c5c886a0c37c 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (6m) 11s ago 16m 439M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (7m) 25s ago 18m 495M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (6m) 25s ago 18m 62.4M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (6m) 11s ago 16m 48.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 da050285f124 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (6m) 25s ago 17m 54.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 0d3d1a65e2c9 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (63s) 25s ago 13m 7977k - 1.5.0 0da6a335fe13 5f8842670a3c 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (57s) 11s ago 13m 8028k - 1.5.0 0da6a335fe13 1f9c124dcc9a 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (5m) 25s ago 16m 65.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 4c456435000b 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (4m) 25s ago 16m 58.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2ff250ddb1d9 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (4m) 25s ago 15m 58.9M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7899f3c3c2f4 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (3m) 25s ago 15m 63.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 94b84aca13ce 2024-09-17T09:34:35.089 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (3m) 11s ago 15m 56.7M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2306402255ea 2024-09-17T09:34:35.090 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (3m) 11s ago 14m 51.1M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0312d071d90b 2024-09-17T09:34:35.090 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (2m) 11s ago 14m 60.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 a5355c784b3c 2024-09-17T09:34:35.090 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (2m) 11s ago 14m 50.7M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 02f285335edf 2024-09-17T09:34:35.090 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 running (43s) 11s ago 13m 35.7M - 2.43.0 a07b618ecd1d f332a1331776 2024-09-17T09:34:35.090 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (105s) 25s ago 12m 77.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 697c12a1dd89 2024-09-17T09:34:35.090 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (95s) 11s ago 12m 74.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 6b0ca358485b 2024-09-17T09:34:35.090 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (100s) 25s ago 11m 75.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 1b791ec24040 2024-09-17T09:34:35.090 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (89s) 11s ago 12m 75.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 ddcc313b787c 2024-09-17T09:34:35.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:35 smithi029 ceph-mon[60489]: from='client.55168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:35.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:35 smithi029 ceph-mon[60489]: pgmap v289: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:35.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:35 smithi029 ceph-mon[58326]: from='client.55168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:35.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:35 smithi029 ceph-mon[58326]: pgmap v289: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:35.740 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph versions' 2024-09-17T09:34:35.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:35 smithi103 ceph-mon[49967]: from='client.55168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:34:35.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:35 smithi103 ceph-mon[49967]: pgmap v289: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:37.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:37 smithi029 ceph-mon[58326]: pgmap v290: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:37.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:37 smithi029 ceph-mon[60489]: pgmap v290: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:37.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:37 smithi103 ceph-mon[49967]: pgmap v290: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:37.836 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:34:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:34:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:34:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:34:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:34:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:34:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:34:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:34:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T09:34:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:34:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:34:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T09:34:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:34:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:34:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 17 2024-09-17T09:34:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:34:37.838 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:34:38.526 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:38 smithi029 ceph-mon[58326]: from='client.? 172.21.15.29:0/393683995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:38.526 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:38 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:38.527 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:38 smithi029 ceph-mon[60489]: from='client.? 172.21.15.29:0/393683995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:38.527 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:38 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:38.656 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-09-17T09:34:38.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:38 smithi103 ceph-mon[49967]: from='client.? 172.21.15.29:0/393683995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:34:38.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:38 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:39.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:39 smithi029 ceph-mon[58326]: pgmap v291: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:39.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:39 smithi029 ceph-mon[60489]: pgmap v291: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:39.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:39 smithi103 ceph-mon[49967]: pgmap v291: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:40.600 INFO:teuthology.orchestra.run.smithi029.stdout:wait for servicemap items w/ changing names to refresh 2024-09-17T09:34:41.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'sleep 60' 2024-09-17T09:34:41.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:41 smithi029 ceph-mon[58326]: pgmap v292: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:41.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:41 smithi029 ceph-mon[60489]: pgmap v292: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:41.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:41 smithi103 ceph-mon[49967]: pgmap v292: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:43.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:43 smithi103 ceph-mon[49967]: pgmap v293: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:43.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:43 smithi029 ceph-mon[60489]: pgmap v293: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:43.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:43 smithi029 ceph-mon[58326]: pgmap v293: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:45.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:45 smithi103 ceph-mon[49967]: pgmap v294: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:45.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:45 smithi029 ceph-mon[58326]: pgmap v294: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:45.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:45 smithi029 ceph-mon[60489]: pgmap v294: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:47.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:47 smithi103 ceph-mon[49967]: pgmap v295: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:47.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:47 smithi029 ceph-mon[58326]: pgmap v295: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:47.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:47 smithi029 ceph-mon[60489]: pgmap v295: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:48.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:48 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:48.549 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:48 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:48.657 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:48 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:49.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:49 smithi029 ceph-mon[58326]: pgmap v296: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:49.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:49 smithi029 ceph-mon[60489]: pgmap v296: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:49.669 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:49 smithi103 ceph-mon[49967]: pgmap v296: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:51.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:51 smithi103 ceph-mon[49967]: pgmap v297: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:51.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:51 smithi029 ceph-mon[58326]: pgmap v297: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:51.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:51 smithi029 ceph-mon[60489]: pgmap v297: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:53.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:53 smithi103 ceph-mon[49967]: pgmap v298: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:53.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:53 smithi029 ceph-mon[58326]: pgmap v298: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:53.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:53 smithi029 ceph-mon[60489]: pgmap v298: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:55.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:55 smithi103 ceph-mon[49967]: pgmap v299: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:55.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:55 smithi029 ceph-mon[58326]: pgmap v299: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:55.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:55 smithi029 ceph-mon[60489]: pgmap v299: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:57.601 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:57 smithi029 ceph-mon[58326]: pgmap v300: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:57.601 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:57 smithi029 ceph-mon[60489]: pgmap v300: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:57.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:57 smithi103 ceph-mon[49967]: pgmap v300: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:34:58.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:58 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:58.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:58 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:58.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:58 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:34:59.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:34:59 smithi029 ceph-mon[58326]: pgmap v301: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:59.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:34:59 smithi029 ceph-mon[60489]: pgmap v301: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:34:59.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:34:59 smithi103 ceph-mon[49967]: pgmap v301: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:01.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:01 smithi103 ceph-mon[49967]: pgmap v302: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:01.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:01 smithi029 ceph-mon[60489]: pgmap v302: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:01.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:01 smithi029 ceph-mon[58326]: pgmap v302: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:03.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:03 smithi103 ceph-mon[49967]: pgmap v303: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:03.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:03 smithi029 ceph-mon[58326]: pgmap v303: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:03.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:03 smithi029 ceph-mon[60489]: pgmap v303: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:05.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:05 smithi103 ceph-mon[49967]: pgmap v304: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:05.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:05 smithi029 ceph-mon[58326]: pgmap v304: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:05.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:05 smithi029 ceph-mon[60489]: pgmap v304: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:07.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:07 smithi103 ceph-mon[49967]: pgmap v305: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:07.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:07 smithi029 ceph-mon[58326]: pgmap v305: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:07.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:07 smithi029 ceph-mon[60489]: pgmap v305: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:08.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:08 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:08.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:08 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:08.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:08 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:09.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:09 smithi103 ceph-mon[49967]: pgmap v306: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:09.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:09 smithi029 ceph-mon[58326]: pgmap v306: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:09.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:09 smithi029 ceph-mon[60489]: pgmap v306: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:11.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:11 smithi103 ceph-mon[49967]: pgmap v307: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:11.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:11 smithi029 ceph-mon[58326]: pgmap v307: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:11.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:11 smithi029 ceph-mon[60489]: pgmap v307: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:13.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:13 smithi103 ceph-mon[49967]: pgmap v308: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:13.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:13 smithi029 ceph-mon[58326]: pgmap v308: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:13.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:13 smithi029 ceph-mon[60489]: pgmap v308: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:15.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:15 smithi103 ceph-mon[49967]: pgmap v309: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:15.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:15 smithi029 ceph-mon[58326]: pgmap v309: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:15.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:15 smithi029 ceph-mon[60489]: pgmap v309: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:17.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:17 smithi103 ceph-mon[49967]: pgmap v310: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:17.800 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:17 smithi029 ceph-mon[58326]: pgmap v310: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:17.800 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:17 smithi029 ceph-mon[60489]: pgmap v310: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:18.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:18 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:18.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:18 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:18.657 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:18 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:19.669 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:19 smithi103 ceph-mon[49967]: pgmap v311: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:19.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:19 smithi029 ceph-mon[58326]: pgmap v311: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:19.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:19 smithi029 ceph-mon[60489]: pgmap v311: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:21.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:21 smithi103 ceph-mon[49967]: pgmap v312: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:21.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:21 smithi029 ceph-mon[58326]: pgmap v312: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:21.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:21 smithi029 ceph-mon[60489]: pgmap v312: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:23.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:23 smithi103 ceph-mon[49967]: pgmap v313: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:23.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:23 smithi029 ceph-mon[58326]: pgmap v313: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:23.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:23 smithi029 ceph-mon[60489]: pgmap v313: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:25.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:25 smithi103 ceph-mon[49967]: pgmap v314: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:25.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:25 smithi029 ceph-mon[58326]: pgmap v314: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:25.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:25 smithi029 ceph-mon[60489]: pgmap v314: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:27.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:27 smithi029 ceph-mon[58326]: pgmap v315: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:27.601 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:27 smithi029 ceph-mon[60489]: pgmap v315: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:27.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:27 smithi103 ceph-mon[49967]: pgmap v315: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:28.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:28 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:28.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:35:28.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:35:28.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:28 smithi103 ceph-mon[49967]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:35:28.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:28 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:28.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:35:28.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:35:28.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:28 smithi029 ceph-mon[58326]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:35:28.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:28 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:28.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T09:35:28.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T09:35:28.799 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:28 smithi029 ceph-mon[60489]: from='mgr.25264 172.21.15.29:0/1357248475' entity='mgr.y' 2024-09-17T09:35:29.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:29 smithi029 ceph-mon[58326]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:35:29.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:29 smithi029 ceph-mon[58326]: pgmap v316: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:29 smithi029 ceph-mon[60489]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:35:29.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:29 smithi029 ceph-mon[60489]: pgmap v316: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:29.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:29 smithi103 ceph-mon[49967]: unable to calc client keyring client.admin placement PlacementSpec(label='_admin'): Cannot place : No matching hosts for label _admin 2024-09-17T09:35:29.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:29 smithi103 ceph-mon[49967]: pgmap v316: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:31.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:31 smithi103 ceph-mon[49967]: pgmap v317: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:31.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:31 smithi029 ceph-mon[58326]: pgmap v317: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:31.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:31 smithi029 ceph-mon[60489]: pgmap v317: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:33.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:33 smithi103 ceph-mon[49967]: pgmap v318: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:33.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:33 smithi029 ceph-mon[58326]: pgmap v318: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:33.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:33 smithi029 ceph-mon[60489]: pgmap v318: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:35.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:35 smithi103 ceph-mon[49967]: pgmap v319: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:35.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:35 smithi029 ceph-mon[58326]: pgmap v319: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:35.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:35 smithi029 ceph-mon[60489]: pgmap v319: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:37.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:37 smithi103 ceph-mon[49967]: pgmap v320: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:37.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:37 smithi029 ceph-mon[58326]: pgmap v320: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:37.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:37 smithi029 ceph-mon[60489]: pgmap v320: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:38.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:38 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:38.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:38 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:38.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:38 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:39.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:39 smithi029 ceph-mon[60489]: pgmap v321: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:39.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:39 smithi029 ceph-mon[58326]: pgmap v321: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:39.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:39 smithi103 ceph-mon[49967]: pgmap v321: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:41.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:41 smithi103 ceph-mon[49967]: pgmap v322: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:41.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:41 smithi029 ceph-mon[58326]: pgmap v322: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:41.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:41 smithi029 ceph-mon[60489]: pgmap v322: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:43.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:43 smithi029 ceph-mon[58326]: pgmap v323: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:43.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:43 smithi029 ceph-mon[60489]: pgmap v323: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:43.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch ps' 2024-09-17T09:35:43.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:43 smithi103 ceph-mon[49967]: pgmap v323: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:45.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:45 smithi029 ceph-mon[58326]: pgmap v324: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:45.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:45 smithi029 ceph-mon[60489]: pgmap v324: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:45.707 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T09:35:45.708 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 running (100s) 95s ago 14m 16.4M - 0.25.0 c8568f914cd2 e3a985039ca0 2024-09-17T09:35:45.708 INFO:teuthology.orchestra.run.smithi029.stdout:grafana.a smithi103 *:3000 running (85s) 81s ago 13m 27.4M - 9.4.7 954c08fa6188 0053793acd87 2024-09-17T09:35:45.708 INFO:teuthology.orchestra.run.smithi029.stdout:iscsi.foo.smithi029.gellkh smithi029 running (2m) 95s ago 13m 50.6M - 3.8 980a0bfcd684 c5c886a0c37c 2024-09-17T09:35:45.708 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi103 *:8443,8765 running (8m) 81s ago 17m 439M - 18.2.4-855-g19e2a2be 980a0bfcd684 88340aeba3ba 2024-09-17T09:35:45.708 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:8443,8765 running (8m) 95s ago 19m 495M - 18.2.4-855-g19e2a2be 980a0bfcd684 29f7e5322eb6 2024-09-17T09:35:45.708 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (7m) 95s ago 19m 62.4M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 88689aaf0f3f 2024-09-17T09:35:45.708 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi103 running (7m) 81s ago 18m 48.0M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 da050285f124 2024-09-17T09:35:45.708 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi029 running (7m) 95s ago 18m 54.7M 2048M 18.2.4-855-g19e2a2be 980a0bfcd684 0d3d1a65e2c9 2024-09-17T09:35:45.708 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.a smithi029 *:9100 running (2m) 95s ago 14m 7977k - 1.5.0 0da6a335fe13 5f8842670a3c 2024-09-17T09:35:45.708 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter.b smithi103 *:9100 running (2m) 81s ago 14m 8028k - 1.5.0 0da6a335fe13 1f9c124dcc9a 2024-09-17T09:35:45.708 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (6m) 95s ago 17m 65.0M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 4c456435000b 2024-09-17T09:35:45.709 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi029 running (5m) 95s ago 17m 58.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2ff250ddb1d9 2024-09-17T09:35:45.709 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi029 running (5m) 95s ago 16m 58.9M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 7899f3c3c2f4 2024-09-17T09:35:45.709 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 smithi029 running (5m) 95s ago 16m 63.4M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 94b84aca13ce 2024-09-17T09:35:45.709 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 smithi103 running (4m) 81s ago 16m 56.7M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 2306402255ea 2024-09-17T09:35:45.709 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 smithi103 running (4m) 81s ago 16m 51.1M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 0312d071d90b 2024-09-17T09:35:45.709 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 smithi103 running (3m) 81s ago 15m 60.8M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 a5355c784b3c 2024-09-17T09:35:45.709 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 smithi103 running (3m) 81s ago 15m 50.7M 4096M 18.2.4-855-g19e2a2be 980a0bfcd684 02f285335edf 2024-09-17T09:35:45.709 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi103 *:9095 running (114s) 81s ago 14m 35.7M - 2.43.0 a07b618ecd1d f332a1331776 2024-09-17T09:35:45.709 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi029.tyykln smithi029 *:8000 running (2m) 95s ago 13m 77.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 697c12a1dd89 2024-09-17T09:35:45.709 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.smithi103.xonepa smithi103 *:8000 running (2m) 81s ago 13m 74.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 6b0ca358485b 2024-09-17T09:35:45.709 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi029.prstdw smithi029 *:80 running (2m) 95s ago 13m 75.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 1b791ec24040 2024-09-17T09:35:45.709 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.smpl.smithi103.gwwbkl smithi103 *:80 running (2m) 81s ago 13m 75.3M - 18.2.4-855-g19e2a2be 980a0bfcd684 ddcc313b787c 2024-09-17T09:35:45.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:45 smithi103 ceph-mon[49967]: pgmap v324: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:46.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph versions' 2024-09-17T09:35:46.701 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:46 smithi029 ceph-mon[58326]: from='client.45297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:35:46.702 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:46 smithi029 ceph-mon[60489]: from='client.45297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:35:46.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:46 smithi103 ceph-mon[49967]: from='client.45297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:35:47.690 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:47 smithi029 ceph-mon[58326]: pgmap v325: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:47.690 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:47 smithi029 ceph-mon[60489]: pgmap v325: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:47.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:47 smithi103 ceph-mon[49967]: pgmap v325: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:48.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:48 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:48.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:48 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:48.581 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:35:48.581 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": { 2024-09-17T09:35:48.581 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 3 2024-09-17T09:35:48.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:35:48.582 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": { 2024-09-17T09:35:48.582 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 2 2024-09-17T09:35:48.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:35:48.582 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": { 2024-09-17T09:35:48.582 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 8 2024-09-17T09:35:48.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:35:48.582 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": { 2024-09-17T09:35:48.582 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 4 2024-09-17T09:35:48.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-17T09:35:48.583 INFO:teuthology.orchestra.run.smithi029.stdout: "overall": { 2024-09-17T09:35:48.583 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)": 17 2024-09-17T09:35:48.583 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-17T09:35:48.583 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:35:48.657 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:48 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:49.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch upgrade status' 2024-09-17T09:35:49.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:49 smithi029 ceph-mon[60489]: from='client.55186 172.21.15.29:0/2904615738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:35:49.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:49 smithi029 ceph-mon[60489]: pgmap v326: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:49.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:49 smithi029 ceph-mon[58326]: from='client.55186 172.21.15.29:0/2904615738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:35:49.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:49 smithi029 ceph-mon[58326]: pgmap v326: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:49.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:49 smithi103 ceph-mon[49967]: from='client.55186 172.21.15.29:0/2904615738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:35:49.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:49 smithi103 ceph-mon[49967]: pgmap v326: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:51.350 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:35:51.350 INFO:teuthology.orchestra.run.smithi029.stdout: "target_image": null, 2024-09-17T09:35:51.350 INFO:teuthology.orchestra.run.smithi029.stdout: "in_progress": false, 2024-09-17T09:35:51.351 INFO:teuthology.orchestra.run.smithi029.stdout: "which": "", 2024-09-17T09:35:51.351 INFO:teuthology.orchestra.run.smithi029.stdout: "services_complete": [], 2024-09-17T09:35:51.351 INFO:teuthology.orchestra.run.smithi029.stdout: "progress": null, 2024-09-17T09:35:51.351 INFO:teuthology.orchestra.run.smithi029.stdout: "message": "", 2024-09-17T09:35:51.351 INFO:teuthology.orchestra.run.smithi029.stdout: "is_paused": false 2024-09-17T09:35:51.351 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:35:51.363 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:51 smithi029 ceph-mon[60489]: pgmap v327: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:51.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:51 smithi029 ceph-mon[58326]: pgmap v327: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:51.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:51 smithi103 ceph-mon[49967]: pgmap v327: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:51.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph health detail' 2024-09-17T09:35:52.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:52 smithi029 ceph-mon[60489]: from='client.55192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:35:52.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:52 smithi029 ceph-mon[58326]: from='client.55192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:35:52.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:52 smithi103 ceph-mon[49967]: from='client.55192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:35:53.502 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:53 smithi029 ceph-mon[60489]: pgmap v328: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:53.503 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:53 smithi029 ceph-mon[58326]: pgmap v328: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:53.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:53 smithi103 ceph-mon[49967]: pgmap v328: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:54.038 INFO:teuthology.orchestra.run.smithi029.stdout:HEALTH_OK 2024-09-17T09:35:54.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:54 smithi029 ceph-mon[58326]: from='client.55198 172.21.15.29:0/1900332631' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:35:54.613 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:54 smithi029 ceph-mon[60489]: from='client.55198 172.21.15.29:0/1900332631' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:35:54.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-09-17T09:35:54.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:54 smithi103 ceph-mon[49967]: from='client.55198 172.21.15.29:0/1900332631' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T09:35:55.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:55 smithi029 ceph-mon[58326]: pgmap v329: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:55.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:55 smithi029 ceph-mon[60489]: pgmap v329: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:55.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:55 smithi103 ceph-mon[49967]: pgmap v329: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:56.759 INFO:teuthology.orchestra.run.smithi029.stdout:true 2024-09-17T09:35:57.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:56 smithi029 ceph-mon[58326]: from='client.55204 172.21.15.29:0/3152418119' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:35:57.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:56 smithi029 ceph-mon[60489]: from='client.55204 172.21.15.29:0/3152418119' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:35:57.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:56 smithi103 ceph-mon[49967]: from='client.55204 172.21.15.29:0/3152418119' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:35:57.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-09-17T09:35:58.038 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:57 smithi029 ceph-mon[58326]: pgmap v330: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:58.038 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:57 smithi029 ceph-mon[60489]: pgmap v330: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:58.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:57 smithi103 ceph-mon[49967]: pgmap v330: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:35:59.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:58 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:59.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:58 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:59.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:58 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:35:59.568 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph version 18.2.4-855-g19e2a2be (19e2a2beb6f41a3ab62650e38614eb29b59dcbb9) reef (stable)" 2024-09-17T09:35:59.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:59 smithi029 ceph-mon[58326]: pgmap v331: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:59.864 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:35:59 smithi029 ceph-mon[58326]: from='client.55210 172.21.15.29:0/1286850767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:35:59.864 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:59 smithi029 ceph-mon[60489]: pgmap v331: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:35:59.864 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:35:59 smithi029 ceph-mon[60489]: from='client.55210 172.21.15.29:0/1286850767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:36:00.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -e sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-09-17T09:36:00.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:59 smithi103 ceph-mon[49967]: pgmap v331: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:36:00.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:35:59 smithi103 ceph-mon[49967]: from='client.55210 172.21.15.29:0/1286850767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T09:36:01.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:01 smithi029 ceph-mon[58326]: pgmap v332: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:36:01.654 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:01 smithi029 ceph-mon[60489]: pgmap v332: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:36:01.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:01 smithi103 ceph-mon[49967]: pgmap v332: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:36:02.245 INFO:teuthology.orchestra.run.smithi029.stdout:osd 8 112s ago - 2024-09-17T09:36:02.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:02 smithi029 ceph-mon[58326]: from='client.55216 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:36:02.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:02 smithi029 ceph-mon[60489]: from='client.55216 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:36:02.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:02 smithi103 ceph-mon[49967]: from='client.55216 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:36:02.913 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T09:36:02.924 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-09-17T09:36:02.924 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-09-17T09:36:03.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:03 smithi029 ceph-mon[58326]: pgmap v333: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:36:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:03 smithi029 ceph-mon[60489]: pgmap v333: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:36:03.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:03 smithi103 ceph-mon[49967]: pgmap v333: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:36:05.533 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-17T09:36:05.534 INFO:teuthology.orchestra.run.smithi029.stdout: "image": "quay.io/ceph/ceph", 2024-09-17T09:36:05.534 INFO:teuthology.orchestra.run.smithi029.stdout: "registry": "quay.io", 2024-09-17T09:36:05.534 INFO:teuthology.orchestra.run.smithi029.stdout: "bare_image": "ceph/ceph", 2024-09-17T09:36:05.534 INFO:teuthology.orchestra.run.smithi029.stdout: "versions": [ 2024-09-17T09:36:05.534 INFO:teuthology.orchestra.run.smithi029.stdout: "19.1.1", 2024-09-17T09:36:05.534 INFO:teuthology.orchestra.run.smithi029.stdout: "19.1.0", 2024-09-17T09:36:05.534 INFO:teuthology.orchestra.run.smithi029.stdout: "18.2.4", 2024-09-17T09:36:05.534 INFO:teuthology.orchestra.run.smithi029.stdout: "18.2.2", 2024-09-17T09:36:05.534 INFO:teuthology.orchestra.run.smithi029.stdout: "18.2.1", 2024-09-17T09:36:05.534 INFO:teuthology.orchestra.run.smithi029.stdout: "18.2.0" 2024-09-17T09:36:05.534 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-17T09:36:05.534 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-17T09:36:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:05 smithi103 ceph-mon[49967]: from='client.45339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:36:05.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:05 smithi103 ceph-mon[49967]: pgmap v334: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:36:05.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:05 smithi029 ceph-mon[58326]: from='client.45339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:36:05.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:05 smithi029 ceph-mon[58326]: pgmap v334: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:36:05.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:05 smithi029 ceph-mon[60489]: from='client.45339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:36:05.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:05 smithi029 ceph-mon[60489]: pgmap v334: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:36:06.137 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-09-17T09:36:07.402 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:07 smithi029 ceph-mon[58326]: pgmap v335: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:36:07.402 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:07 smithi029 ceph-mon[60489]: pgmap v335: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:36:07.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:07 smithi103 ceph-mon[49967]: pgmap v335: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T09:36:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "16.2.0", 2024-09-17T09:36:08.427 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:08 smithi029 ceph-mon[58326]: from='client.55228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:36:08.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:08 smithi029 ceph-mon[58326]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:36:08.428 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:08 smithi029 ceph-mon[60489]: from='client.55228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:36:08.428 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:08 smithi029 ceph-mon[60489]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:36:08.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:08 smithi103 ceph-mon[49967]: from='client.55228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:36:08.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:08 smithi103 ceph-mon[49967]: from='client.45021 -' entity='client.iscsi.foo.smithi029.gellkh' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T09:36:09.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-09-17T09:36:09.600 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:09 smithi029 ceph-mon[58326]: pgmap v336: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:36:09.600 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:09 smithi029 ceph-mon[60489]: pgmap v336: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:36:09.755 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:09 smithi103 ceph-mon[49967]: pgmap v336: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:36:11.597 INFO:teuthology.orchestra.run.smithi029.stdout: "v16.2.2", 2024-09-17T09:36:11.598 INFO:teuthology.orchestra.run.smithi029.stdout: "v16.2.2-20210505", 2024-09-17T09:36:11.598 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:11 smithi029 ceph-mon[60489]: from='client.55234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:36:11.598 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:11 smithi029 ceph-mon[60489]: pgmap v337: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:36:11.598 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:11 smithi029 ceph-mon[58326]: from='client.55234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:36:11.598 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:11 smithi029 ceph-mon[58326]: pgmap v337: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:36:11.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:11 smithi103 ceph-mon[49967]: from='client.55234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T09:36:11.756 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:11 smithi103 ceph-mon[49967]: pgmap v337: 161 pgs: 161 active+clean; 604 KiB data, 321 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T09:36:12.188 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-17T09:36:12.197 INFO:tasks.cephadm:Teardown begin 2024-09-17T09:36:12.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:36:12.228 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:36:12.262 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-17T09:36:12.262 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T09:36:12.285 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T09:36:12.319 INFO:tasks.cephadm:Stopping all daemons... 2024-09-17T09:36:12.319 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-17T09:36:12.320 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.a 2024-09-17T09:36:12.548 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:12 smithi029 systemd[1]: Stopping Ceph mon.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:36:13.013 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-a[58321]: 2024-09-17T09:36:12.639+0000 7f9aa5ace640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:36:13.013 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:12 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-a[58321]: 2024-09-17T09:36:12.639+0000 7f9aa5ace640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-17T09:36:13.013 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:12 smithi029 podman[98040]: 2024-09-17 09:36:12.742617402 +0000 UTC m=+0.222544054 container died 88689aaf0f3f98c46cf717a6e8b89505e12255ebdcc0461e2dd3f3210ede43a4 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-17T09:36:13.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:13 smithi029 podman[98040]: 2024-09-17 09:36:13.1015412 +0000 UTC m=+0.581467806 container cleanup 88689aaf0f3f98c46cf717a6e8b89505e12255ebdcc0461e2dd3f3210ede43a4 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-17T09:36:13.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Sep 17 09:36:13 smithi029 bash[98040]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-a 2024-09-17T09:36:13.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.a.service' 2024-09-17T09:36:13.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:36:13.595 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-17T09:36:13.595 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-17T09:36:13.595 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.c 2024-09-17T09:36:13.798 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:13 smithi029 systemd[1]: Stopping Ceph mon.c for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:36:14.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-c[60485]: 2024-09-17T09:36:13.902+0000 7f676ebb8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:36:14.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-c[60485]: 2024-09-17T09:36:13.902+0000 7f676ebb8640 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-09-17T09:36:14.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:14 smithi029 podman[98162]: 2024-09-17 09:36:14.137605376 +0000 UTC m=+0.350328356 container died 0d3d1a65e2c90bcd7059d6fffa435afdbbb8f998f6048dd6f5c04723ffdc3154 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-17T09:36:14.749 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:14 smithi029 podman[98162]: 2024-09-17 09:36:14.487784321 +0000 UTC m=+0.700507308 container cleanup 0d3d1a65e2c90bcd7059d6fffa435afdbbb8f998f6048dd6f5c04723ffdc3154 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-17T09:36:14.750 INFO:journalctl@ceph.mon.c.smithi029.stdout:Sep 17 09:36:14 smithi029 bash[98162]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-c 2024-09-17T09:36:14.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.c.service' 2024-09-17T09:36:14.961 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:36:14.961 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-17T09:36:14.961 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-17T09:36:14.961 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.b 2024-09-17T09:36:15.256 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:15 smithi103 systemd[1]: Stopping Ceph mon.b for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:36:15.724 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-b[49942]: 2024-09-17T09:36:15.354+0000 7fa2a3ee1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:36:15.724 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-b[49942]: 2024-09-17T09:36:15.354+0000 7fa2a3ee1640 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-09-17T09:36:15.724 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:15 smithi103 podman[80046]: 2024-09-17 09:36:15.436541167 +0000 UTC m=+0.276799187 container died da050285f124fc567176c596942e5fef7d87666a4bd27b293c193dc80cd9accd (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-17T09:36:16.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:15 smithi103 podman[80046]: 2024-09-17 09:36:15.920570747 +0000 UTC m=+0.760828764 container cleanup da050285f124fc567176c596942e5fef7d87666a4bd27b293c193dc80cd9accd (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-17T09:36:16.006 INFO:journalctl@ceph.mon.b.smithi103.stdout:Sep 17 09:36:15 smithi103 bash[80046]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mon-b 2024-09-17T09:36:16.335 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mon.b.service' 2024-09-17T09:36:16.380 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:36:16.381 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-17T09:36:16.381 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-17T09:36:16.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.y 2024-09-17T09:36:16.671 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:36:16 smithi029 systemd[1]: Stopping Ceph mgr.y for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:36:16.991 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:36:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:36:16.669+0000 7fe7980ba640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:36:16.991 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:36:16 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y[53266]: 2024-09-17T09:36:16.669+0000 7fe7980ba640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-17T09:36:16.991 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:36:16 smithi029 podman[98284]: 2024-09-17 09:36:16.70109014 +0000 UTC m=+0.122114676 container died 29f7e5322eb6da976f12a34a19f60fb356a7097563c3ca8fc67ef9181f7348d7 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-17T09:36:17.298 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:36:17 smithi029 podman[98284]: 2024-09-17 09:36:17.098482109 +0000 UTC m=+0.519506642 container cleanup 29f7e5322eb6da976f12a34a19f60fb356a7097563c3ca8fc67ef9181f7348d7 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:36:17.298 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Sep 17 09:36:17 smithi029 bash[98284]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-y 2024-09-17T09:36:17.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-17T09:36:17.515 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:36:17.515 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-17T09:36:17.515 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-17T09:36:17.516 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.x 2024-09-17T09:36:17.850 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:36:17 smithi103 systemd[1]: Stopping Ceph mgr.x for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:36:18.129 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:36:17 smithi103 podman[80168]: 2024-09-17 09:36:17.849239816 +0000 UTC m=+0.142521111 container died 88340aeba3bad77e143f84fae732ba60582d6bd18d88695c4cb92145f9b2370a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-17T09:36:18.463 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:36:18 smithi103 podman[80168]: 2024-09-17 09:36:18.218343259 +0000 UTC m=+0.511624545 container cleanup 88340aeba3bad77e143f84fae732ba60582d6bd18d88695c4cb92145f9b2370a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b) 2024-09-17T09:36:18.463 INFO:journalctl@ceph.mgr.x.smithi103.stdout:Sep 17 09:36:18 smithi103 bash[80168]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-mgr-x 2024-09-17T09:36:18.602 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-17T09:36:18.637 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:36:18.637 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-17T09:36:18.638 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-17T09:36:18.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.0 2024-09-17T09:36:18.925 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:18 smithi029 systemd[1]: Stopping Ceph osd.0 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:36:19.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0[65033]: 2024-09-17T09:36:18.924+0000 7fe7a57f6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:36:19.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0[65033]: 2024-09-17T09:36:18.924+0000 7fe7a57f6640 -1 osd.0 143 *** Got signal Terminated *** 2024-09-17T09:36:19.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:18 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0[65033]: 2024-09-17T09:36:18.924+0000 7fe7a57f6640 -1 osd.0 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:36:23.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:36:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:36:23.513Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused" 2024-09-17T09:36:23.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:36:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:36:23.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused" 2024-09-17T09:36:24.549 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:24 smithi029 podman[98408]: 2024-09-17 09:36:24.267689501 +0000 UTC m=+5.433833833 container died 4c456435000b33b3310d9b3cfc32c3eb25af1f800cb692b773dc24f8555c4924 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731) 2024-09-17T09:36:25.016 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:24 smithi029 podman[98408]: 2024-09-17 09:36:24.728781147 +0000 UTC m=+5.894925478 container cleanup 4c456435000b33b3310d9b3cfc32c3eb25af1f800cb692b773dc24f8555c4924 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-09-17T09:36:25.016 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:24 smithi029 bash[98408]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0 2024-09-17T09:36:25.298 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:25 smithi029 podman[98423]: 2024-09-17 09:36:25.015416177 +0000 UTC m=+0.746737070 container remove 4c456435000b33b3310d9b3cfc32c3eb25af1f800cb692b773dc24f8555c4924 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True) 2024-09-17T09:36:25.649 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:25 smithi029 podman[98514]: 2024-09-17 09:36:25.265710353 +0000 UTC m=+0.016392764 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:36:25.650 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:25 smithi029 podman[98514]: 2024-09-17 09:36:25.367503278 +0000 UTC m=+0.118185670 container create 6646fcb666efdf8330a7226e1c667889079da5d6f6838c327039991056697259 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-17T09:36:26.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:25 smithi029 podman[98514]: 2024-09-17 09:36:25.648810488 +0000 UTC m=+0.399492898 container init 6646fcb666efdf8330a7226e1c667889079da5d6f6838c327039991056697259 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-17T09:36:26.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:25 smithi029 podman[98514]: 2024-09-17 09:36:25.652771568 +0000 UTC m=+0.403453969 container start 6646fcb666efdf8330a7226e1c667889079da5d6f6838c327039991056697259 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-17T09:36:26.048 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:25 smithi029 podman[98514]: 2024-09-17 09:36:25.74901474 +0000 UTC m=+0.499697150 container attach 6646fcb666efdf8330a7226e1c667889079da5d6f6838c327039991056697259 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:36:26.049 INFO:journalctl@ceph.osd.0.smithi029.stdout:Sep 17 09:36:25 smithi029 podman[98514]: 2024-09-17 09:36:25.846469435 +0000 UTC m=+0.597151840 container died 6646fcb666efdf8330a7226e1c667889079da5d6f6838c327039991056697259 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2) 2024-09-17T09:36:26.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.0.service' 2024-09-17T09:36:26.479 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:36:26.479 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-17T09:36:26.479 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-17T09:36:26.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.1 2024-09-17T09:36:26.679 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:26 smithi029 systemd[1]: Stopping Ceph osd.1 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:36:26.975 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1[68006]: 2024-09-17T09:36:26.810+0000 7f3302538640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:36:26.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1[68006]: 2024-09-17T09:36:26.810+0000 7f3302538640 -1 osd.1 143 *** Got signal Terminated *** 2024-09-17T09:36:26.976 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:26 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1[68006]: 2024-09-17T09:36:26.810+0000 7f3302538640 -1 osd.1 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:36:32.540 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:32 smithi029 podman[98618]: 2024-09-17 09:36:32.251948586 +0000 UTC m=+5.534567439 container died 2ff250ddb1d9df1dee5642100252b078f51d77b65c1b46cae55128b2ad38ae8f (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:36:32.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:32 smithi029 podman[98618]: 2024-09-17 09:36:32.648291145 +0000 UTC m=+5.930909987 container cleanup 2ff250ddb1d9df1dee5642100252b078f51d77b65c1b46cae55128b2ad38ae8f (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-17T09:36:32.801 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:32 smithi029 bash[98618]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1 2024-09-17T09:36:33.235 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:32 smithi029 podman[98646]: 2024-09-17 09:36:32.892914791 +0000 UTC m=+0.637830735 container remove 2ff250ddb1d9df1dee5642100252b078f51d77b65c1b46cae55128b2ad38ae8f (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-17T09:36:33.509 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:33 smithi029 podman[98732]: 2024-09-17 09:36:33.137606543 +0000 UTC m=+0.017145321 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:36:33.510 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:33 smithi029 podman[98732]: 2024-09-17 09:36:33.269869886 +0000 UTC m=+0.149408667 container create 8f8ddfe7414db646d21935bee377a8d0bf3274698d56e1bb7b1cc0200a417fbe (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-17T09:36:33.510 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:33 smithi029 podman[98732]: 2024-09-17 09:36:33.508884043 +0000 UTC m=+0.388422821 container init 8f8ddfe7414db646d21935bee377a8d0bf3274698d56e1bb7b1cc0200a417fbe (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2) 2024-09-17T09:36:33.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:36:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:36:33.513Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=9 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused" 2024-09-17T09:36:33.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:36:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:36:33.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused" 2024-09-17T09:36:33.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:36:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:36:33.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused" 2024-09-17T09:36:33.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:33 smithi029 podman[98732]: 2024-09-17 09:36:33.512348814 +0000 UTC m=+0.391887599 container start 8f8ddfe7414db646d21935bee377a8d0bf3274698d56e1bb7b1cc0200a417fbe (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-17T09:36:33.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:33 smithi029 podman[98732]: 2024-09-17 09:36:33.601116208 +0000 UTC m=+0.480654996 container attach 8f8ddfe7414db646d21935bee377a8d0bf3274698d56e1bb7b1cc0200a417fbe (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True) 2024-09-17T09:36:33.799 INFO:journalctl@ceph.osd.1.smithi029.stdout:Sep 17 09:36:33 smithi029 podman[98732]: 2024-09-17 09:36:33.708553707 +0000 UTC m=+0.588092494 container died 8f8ddfe7414db646d21935bee377a8d0bf3274698d56e1bb7b1cc0200a417fbe (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-17T09:36:34.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.1.service' 2024-09-17T09:36:34.416 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:36:34.417 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-17T09:36:34.417 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-17T09:36:34.417 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.2 2024-09-17T09:36:34.706 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:34 smithi029 systemd[1]: Stopping Ceph osd.2 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:36:35.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2[70908]: 2024-09-17T09:36:34.704+0000 7fd6c95f6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:36:35.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2[70908]: 2024-09-17T09:36:34.704+0000 7fd6c95f6640 -1 osd.2 143 *** Got signal Terminated *** 2024-09-17T09:36:35.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:34 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2[70908]: 2024-09-17T09:36:34.704+0000 7fd6c95f6640 -1 osd.2 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:36:40.426 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:40 smithi029 podman[98836]: 2024-09-17 09:36:40.139179197 +0000 UTC m=+5.534205668 container died 7899f3c3c2f4823a0c652fd0ce241da3101e6d219cfa1e66b4463ee8d7eb8a4f (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-17T09:36:40.771 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:40 smithi029 podman[98836]: 2024-09-17 09:36:40.591042351 +0000 UTC m=+5.986068821 container cleanup 7899f3c3c2f4823a0c652fd0ce241da3101e6d219cfa1e66b4463ee8d7eb8a4f (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-17T09:36:40.771 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:40 smithi029 bash[98836]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2 2024-09-17T09:36:41.048 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:40 smithi029 podman[98855]: 2024-09-17 09:36:40.770328181 +0000 UTC m=+0.622028393 container remove 7899f3c3c2f4823a0c652fd0ce241da3101e6d219cfa1e66b4463ee8d7eb8a4f (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-17T09:36:41.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:41 smithi029 podman[98942]: 2024-09-17 09:36:41.205213063 +0000 UTC m=+0.105816191 container create c135d3dd641ce06bb72b1ef0cea24cad0e49011d9603e7e25f100984cf245751 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-17T09:36:41.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:41 smithi029 podman[98942]: 2024-09-17 09:36:41.115931043 +0000 UTC m=+0.016534180 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:36:41.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:41 smithi029 podman[98942]: 2024-09-17 09:36:41.442807928 +0000 UTC m=+0.343411073 container init c135d3dd641ce06bb72b1ef0cea24cad0e49011d9603e7e25f100984cf245751 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-17T09:36:41.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:41 smithi029 podman[98942]: 2024-09-17 09:36:41.446241971 +0000 UTC m=+0.346845100 container start c135d3dd641ce06bb72b1ef0cea24cad0e49011d9603e7e25f100984cf245751 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2) 2024-09-17T09:36:41.527 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:41 smithi029 podman[98942]: 2024-09-17 09:36:41.526309023 +0000 UTC m=+0.426912151 container attach c135d3dd641ce06bb72b1ef0cea24cad0e49011d9603e7e25f100984cf245751 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-17T09:36:41.798 INFO:journalctl@ceph.osd.2.smithi029.stdout:Sep 17 09:36:41 smithi029 podman[98942]: 2024-09-17 09:36:41.638054822 +0000 UTC m=+0.538657955 container died c135d3dd641ce06bb72b1ef0cea24cad0e49011d9603e7e25f100984cf245751 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-17T09:36:42.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.2.service' 2024-09-17T09:36:42.280 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:36:42.280 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-17T09:36:42.280 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-17T09:36:42.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.3 2024-09-17T09:36:42.548 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:42 smithi029 systemd[1]: Stopping Ceph osd.3 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:36:43.048 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3[73469]: 2024-09-17T09:36:42.638+0000 7f06a4312640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:36:43.048 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3[73469]: 2024-09-17T09:36:42.638+0000 7f06a4312640 -1 osd.3 143 *** Got signal Terminated *** 2024-09-17T09:36:43.048 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:42 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3[73469]: 2024-09-17T09:36:42.638+0000 7f06a4312640 -1 osd.3 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:36:43.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:36:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:36:43.514Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=9 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused" 2024-09-17T09:36:43.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:36:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:36:43.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused" 2024-09-17T09:36:43.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:36:43 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:36:43.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused" 2024-09-17T09:36:45.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:44 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:44.839+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:46.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:45 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:45.827+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:47.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:46 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:46.781+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:47.548 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:47 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3[73469]: 2024-09-17T09:36:47.290+0000 7f069c91e640 -1 osd.3 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.738173+0000 front 2024-09-17T09:36:23.738190+0000 (oldest deadline 2024-09-17T09:36:47.238295+0000) 2024-09-17T09:36:48.006 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:36:47.708+0000 7f95ab395640 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:21.551470+0000 front 2024-09-17T09:36:21.551479+0000 (oldest deadline 2024-09-17T09:36:47.451204+0000) 2024-09-17T09:36:48.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:47 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:47.784+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:48.338 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:48.234+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:36:48.517 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:48 smithi029 podman[99043]: 2024-09-17 09:36:48.171486456 +0000 UTC m=+5.655039826 container died 94b84aca13ce200a1801bf455dbf6a3454f469efcaad660f30530d743f466865 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-17T09:36:48.667 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:48.336+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:36:48.795 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:48 smithi029 podman[99043]: 2024-09-17 09:36:48.516837451 +0000 UTC m=+6.000390830 container cleanup 94b84aca13ce200a1801bf455dbf6a3454f469efcaad660f30530d743f466865 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-09-17T09:36:48.795 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:48 smithi029 bash[99043]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3 2024-09-17T09:36:49.005 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:36:48.665+0000 7f95ab395640 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:21.551470+0000 front 2024-09-17T09:36:21.551479+0000 (oldest deadline 2024-09-17T09:36:47.451204+0000) 2024-09-17T09:36:49.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:48 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:48.752+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:49.048 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:48 smithi029 podman[99060]: 2024-09-17 09:36:48.794395907 +0000 UTC m=+0.622896194 container remove 94b84aca13ce200a1801bf455dbf6a3454f469efcaad660f30530d743f466865 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:36:49.478 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:49 smithi029 podman[99147]: 2024-09-17 09:36:49.050077531 +0000 UTC m=+0.016515484 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:36:49.478 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:49 smithi029 podman[99147]: 2024-09-17 09:36:49.155901443 +0000 UTC m=+0.122339601 container create 0a459e54361dd22af5e3b4a1de30c2e1a1c5fe09ccaa9ab1e943ce45531ea0a7 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-09-17T09:36:49.478 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:49 smithi029 podman[99147]: 2024-09-17 09:36:49.384978811 +0000 UTC m=+0.351416748 container init 0a459e54361dd22af5e3b4a1de30c2e1a1c5fe09ccaa9ab1e943ce45531ea0a7 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:36:49.478 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:49 smithi029 podman[99147]: 2024-09-17 09:36:49.38854774 +0000 UTC m=+0.354985682 container start 0a459e54361dd22af5e3b4a1de30c2e1a1c5fe09ccaa9ab1e943ce45531ea0a7 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:36:49.478 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:49 smithi029 podman[99147]: 2024-09-17 09:36:49.476975647 +0000 UTC m=+0.443413588 container attach 0a459e54361dd22af5e3b4a1de30c2e1a1c5fe09ccaa9ab1e943ce45531ea0a7 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-17T09:36:49.505 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:49.237+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:36:49.506 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:49.384+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:36:49.798 INFO:journalctl@ceph.osd.3.smithi029.stdout:Sep 17 09:36:49 smithi029 podman[99147]: 2024-09-17 09:36:49.592197514 +0000 UTC m=+0.558635451 container died 0a459e54361dd22af5e3b4a1de30c2e1a1c5fe09ccaa9ab1e943ce45531ea0a7 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-17T09:36:50.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:49.771+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:50.006 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:49 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:36:49.634+0000 7f95ab395640 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:21.551470+0000 front 2024-09-17T09:36:21.551479+0000 (oldest deadline 2024-09-17T09:36:47.451204+0000) 2024-09-17T09:36:50.178 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.3.service' 2024-09-17T09:36:50.221 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:36:50.221 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-17T09:36:50.221 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-17T09:36:50.221 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.4 2024-09-17T09:36:50.506 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:50 smithi103 systemd[1]: Stopping Ceph osd.4 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:36:50.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:50.258+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:36:50.506 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:50.401+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:36:51.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:50.776+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:51.006 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:36:50.537+0000 7f95aed81640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:36:51.006 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:36:50.537+0000 7f95aed81640 -1 osd.4 143 *** Got signal Terminated *** 2024-09-17T09:36:51.006 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:36:50.537+0000 7f95aed81640 -1 osd.4 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:36:51.006 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:50 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:36:50.608+0000 7f95ab395640 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:21.551470+0000 front 2024-09-17T09:36:21.551479+0000 (oldest deadline 2024-09-17T09:36:47.451204+0000) 2024-09-17T09:36:51.636 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:51.295+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:36:51.637 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:51.410+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:36:52.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:51.727+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:52.006 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:36:51.635+0000 7f95ab395640 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:21.551470+0000 front 2024-09-17T09:36:21.551479+0000 (oldest deadline 2024-09-17T09:36:47.451204+0000) 2024-09-17T09:36:52.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:36:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:36:51.666Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.29:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.29:8765: connect: connection refused" 2024-09-17T09:36:52.006 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:36:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:36:51.666Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.29:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.29:8765: connect: connection refused" 2024-09-17T09:36:52.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:36:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:36:51.667Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.29:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.29:8765: connect: connection refused" 2024-09-17T09:36:52.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:36:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:36:51.667Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.29:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.29:8765: connect: connection refused" 2024-09-17T09:36:52.007 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:36:51 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:36:51.669Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.29:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.29:8765: connect: connection refused" 2024-09-17T09:36:52.685 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:52.316+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:36:52.685 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:52.432+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:36:53.006 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:36:52.683+0000 7f95ab395640 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:21.551470+0000 front 2024-09-17T09:36:21.551479+0000 (oldest deadline 2024-09-17T09:36:47.451204+0000) 2024-09-17T09:36:53.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:52 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:52.756+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:53.646 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:53.275+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:36:53.647 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:53.398+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:36:53.647 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:53.398+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:36:53.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:36:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:36:53.515Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=9 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused" 2024-09-17T09:36:53.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:36:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:36:53.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused" 2024-09-17T09:36:53.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:36:53 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:36:53.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused" 2024-09-17T09:36:54.005 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:53.709+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:54.006 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:36:53.644+0000 7f95ab395640 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:21.551470+0000 front 2024-09-17T09:36:21.551479+0000 (oldest deadline 2024-09-17T09:36:47.451204+0000) 2024-09-17T09:36:54.006 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:53 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:36:53.644+0000 7f95ab395640 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:27.451636+0000 front 2024-09-17T09:36:27.451574+0000 (oldest deadline 2024-09-17T09:36:52.751564+0000) 2024-09-17T09:36:54.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:54.232+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:36:54.506 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:54.447+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:36:54.506 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:54.447+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:36:55.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:54.723+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:55.006 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:36:54.660+0000 7f95ab395640 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:21.551470+0000 front 2024-09-17T09:36:21.551479+0000 (oldest deadline 2024-09-17T09:36:47.451204+0000) 2024-09-17T09:36:55.006 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:54 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4[53524]: 2024-09-17T09:36:54.660+0000 7f95ab395640 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:27.451636+0000 front 2024-09-17T09:36:27.451574+0000 (oldest deadline 2024-09-17T09:36:52.751564+0000) 2024-09-17T09:36:55.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:55.193+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:36:55.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:55.193+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:29.012987+0000 front 2024-09-17T09:36:29.012878+0000 (oldest deadline 2024-09-17T09:36:54.312970+0000) 2024-09-17T09:36:55.506 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:55.438+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:36:55.506 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:55.438+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:36:56.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:55.731+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:56.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:55 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:55.731+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:36:56.341 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:56 smithi103 podman[80304]: 2024-09-17 09:36:56.049559296 +0000 UTC m=+5.624881006 container died 2306402255eaed03c119220983081f76e5aa258b2e21b327f59e0ef1090c06ea (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-17T09:36:56.341 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:56.148+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:36:56.341 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:56.148+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:29.012987+0000 front 2024-09-17T09:36:29.012878+0000 (oldest deadline 2024-09-17T09:36:54.312970+0000) 2024-09-17T09:36:56.703 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:56.456+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:36:56.704 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:56.456+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:36:56.704 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:56 smithi103 podman[80304]: 2024-09-17 09:36:56.513251549 +0000 UTC m=+6.088573250 container cleanup 2306402255eaed03c119220983081f76e5aa258b2e21b327f59e0ef1090c06ea (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux ) 2024-09-17T09:36:56.704 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:56 smithi103 bash[80304]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4 2024-09-17T09:36:57.005 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:56 smithi103 podman[80321]: 2024-09-17 09:36:56.756861466 +0000 UTC m=+0.701413822 container remove 2306402255eaed03c119220983081f76e5aa258b2e21b327f59e0ef1090c06ea (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4, RELEASE=reef-19e2a2b, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-17T09:36:57.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:56.701+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:57.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:56 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:56.701+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:36:57.353 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:57.127+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:36:57.353 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:57.127+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:29.012987+0000 front 2024-09-17T09:36:29.012878+0000 (oldest deadline 2024-09-17T09:36:54.312970+0000) 2024-09-17T09:36:57.353 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:57 smithi103 podman[80405]: 2024-09-17 09:36:56.990779009 +0000 UTC m=+0.015720851 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:36:57.353 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:57 smithi103 podman[80405]: 2024-09-17 09:36:57.124482535 +0000 UTC m=+0.149424373 container create e0884c78945243e58cf17c10299289ed494751be28be5ffcced71057c4547d81 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-17T09:36:57.353 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:57 smithi103 podman[80405]: 2024-09-17 09:36:57.35190956 +0000 UTC m=+0.376851392 container init e0884c78945243e58cf17c10299289ed494751be28be5ffcced71057c4547d81 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=reef-19e2a2b, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-17T09:36:57.717 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:57.498+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:36:57.717 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:57.498+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:36:57.717 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:57 smithi103 podman[80405]: 2024-09-17 09:36:57.355291147 +0000 UTC m=+0.380232987 container start e0884c78945243e58cf17c10299289ed494751be28be5ffcced71057c4547d81 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=reef-19e2a2b) 2024-09-17T09:36:57.717 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:57 smithi103 podman[80405]: 2024-09-17 09:36:57.443606381 +0000 UTC m=+0.468548219 container attach e0884c78945243e58cf17c10299289ed494751be28be5ffcced71057c4547d81 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=reef-19e2a2b) 2024-09-17T09:36:57.718 INFO:journalctl@ceph.osd.4.smithi103.stdout:Sep 17 09:36:57 smithi103 podman[80405]: 2024-09-17 09:36:57.535346844 +0000 UTC m=+0.560288685 container died e0884c78945243e58cf17c10299289ed494751be28be5ffcced71057c4547d81 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-17T09:36:58.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:57.715+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:58.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:57 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:57.715+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:36:58.151 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.4.service' 2024-09-17T09:36:58.194 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:36:58.194 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-17T09:36:58.194 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-17T09:36:58.194 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.5 2024-09-17T09:36:58.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:58.169+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:36:58.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:58.169+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:29.012987+0000 front 2024-09-17T09:36:29.012878+0000 (oldest deadline 2024-09-17T09:36:54.312970+0000) 2024-09-17T09:36:58.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:58 smithi103 systemd[1]: Stopping Ceph osd.5 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:36:59.006 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:58.513+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:36:59.006 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:58.513+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:36:59.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:58.507+0000 7f56e3e3a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:36:59.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:58.507+0000 7f56e3e3a640 -1 osd.5 143 *** Got signal Terminated *** 2024-09-17T09:36:59.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:58.507+0000 7f56e3e3a640 -1 osd.5 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:36:59.007 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:58.679+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:59.007 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:58 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:58.679+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:36:59.476 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:59.164+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:36:59.476 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:36:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:36:59.164+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:29.012987+0000 front 2024-09-17T09:36:29.012878+0000 (oldest deadline 2024-09-17T09:36:54.312970+0000) 2024-09-17T09:36:59.755 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:59.475+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:36:59.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:36:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:36:59.475+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:36:59.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:59.638+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:36:59.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:36:59 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:36:59.638+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:00.487 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:37:00.176+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:37:00.487 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:37:00.176+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:29.012987+0000 front 2024-09-17T09:36:29.012878+0000 (oldest deadline 2024-09-17T09:36:54.312970+0000) 2024-09-17T09:37:00.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:00.485+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:37:00.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:00.485+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:37:00.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:00.670+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:00.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:00 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:00.670+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:01.474 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:37:01.213+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:37:01.475 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:37:01.213+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:29.012987+0000 front 2024-09-17T09:36:29.012878+0000 (oldest deadline 2024-09-17T09:36:54.312970+0000) 2024-09-17T09:37:01.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:01.473+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:37:01.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:01.473+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:37:01.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:01.699+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:01.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:01.699+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:01.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:01 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:01.699+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:02.491 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:37:02.208+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:37:02.491 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:37:02.208+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:29.012987+0000 front 2024-09-17T09:36:29.012878+0000 (oldest deadline 2024-09-17T09:36:54.312970+0000) 2024-09-17T09:37:02.491 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:37:02.208+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:38.314368+0000 front 2024-09-17T09:36:38.314402+0000 (oldest deadline 2024-09-17T09:37:01.214271+0000) 2024-09-17T09:37:02.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:02.489+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:37:02.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:02.489+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:37:02.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:02.733+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:02.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:02.733+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:02.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:02 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:02.733+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:03.506 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:03.455+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:37:03.506 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:03.455+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:37:03.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:37:03.214+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:22.611976+0000 front 2024-09-17T09:36:22.611890+0000 (oldest deadline 2024-09-17T09:36:47.911849+0000) 2024-09-17T09:37:03.506 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:37:03.214+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:29.012987+0000 front 2024-09-17T09:36:29.012878+0000 (oldest deadline 2024-09-17T09:36:54.312970+0000) 2024-09-17T09:37:03.507 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5[55883]: 2024-09-17T09:37:03.214+0000 7f56e044e640 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:38.314368+0000 front 2024-09-17T09:36:38.314402+0000 (oldest deadline 2024-09-17T09:37:01.214271+0000) 2024-09-17T09:37:03.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:37:03.515Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=9 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused" 2024-09-17T09:37:03.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:37:03.516Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused" 2024-09-17T09:37:03.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:03 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:37:03.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused" 2024-09-17T09:37:04.256 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:04 smithi103 podman[80508]: 2024-09-17 09:37:04.012980332 +0000 UTC m=+5.597201839 container died 0312d071d90be31f4e4585ca386d3d447e80601ec97d6661d27a00de7897d403 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-17T09:37:04.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:03.775+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:04.257 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:03.775+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:04.257 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:03 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:03.775+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:04.754 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:04 smithi103 podman[80508]: 2024-09-17 09:37:04.475328514 +0000 UTC m=+6.059550060 container cleanup 0312d071d90be31f4e4585ca386d3d447e80601ec97d6661d27a00de7897d403 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5, RELEASE=reef-19e2a2b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-17T09:37:04.755 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:04 smithi103 bash[80508]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5 2024-09-17T09:37:04.755 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:04.485+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:37:04.755 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:04.485+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:37:04.755 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:04.485+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:38.132268+0000 front 2024-09-17T09:36:38.132331+0000 (oldest deadline 2024-09-17T09:37:03.932728+0000) 2024-09-17T09:37:05.006 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:04 smithi103 podman[80524]: 2024-09-17 09:37:04.776793485 +0000 UTC m=+0.755950242 container remove 0312d071d90be31f4e4585ca386d3d447e80601ec97d6661d27a00de7897d403 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-17T09:37:05.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:04.753+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:05.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:04.753+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:05.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:04 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:04.753+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:05.406 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:05 smithi103 podman[80613]: 2024-09-17 09:37:04.992517511 +0000 UTC m=+0.015913634 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:37:05.406 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:05 smithi103 podman[80613]: 2024-09-17 09:37:05.111063636 +0000 UTC m=+0.134459747 container create cc9f5b5aacc8b9f32920cd4076ec56927b2243cdd09aeeda64804e0b2495d886 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b) 2024-09-17T09:37:05.406 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:05 smithi103 podman[80613]: 2024-09-17 09:37:05.321769186 +0000 UTC m=+0.345165297 container init cc9f5b5aacc8b9f32920cd4076ec56927b2243cdd09aeeda64804e0b2495d886 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True) 2024-09-17T09:37:05.406 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:05 smithi103 podman[80613]: 2024-09-17 09:37:05.325056152 +0000 UTC m=+0.348452271 container start cc9f5b5aacc8b9f32920cd4076ec56927b2243cdd09aeeda64804e0b2495d886 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b) 2024-09-17T09:37:05.407 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:05 smithi103 podman[80613]: 2024-09-17 09:37:05.405040018 +0000 UTC m=+0.428436152 container attach cc9f5b5aacc8b9f32920cd4076ec56927b2243cdd09aeeda64804e0b2495d886 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b) 2024-09-17T09:37:05.755 INFO:journalctl@ceph.osd.5.smithi103.stdout:Sep 17 09:37:05 smithi103 podman[80613]: 2024-09-17 09:37:05.517541362 +0000 UTC m=+0.540937491 container died cc9f5b5aacc8b9f32920cd4076ec56927b2243cdd09aeeda64804e0b2495d886 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-17T09:37:05.755 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:05.475+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:37:05.755 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:05.475+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:37:05.755 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:05.475+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:38.132268+0000 front 2024-09-17T09:36:38.132331+0000 (oldest deadline 2024-09-17T09:37:03.932728+0000) 2024-09-17T09:37:06.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:05.753+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:06.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:05.753+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:06.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:05 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:05.753+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:06.136 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.5.service' 2024-09-17T09:37:06.173 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:37:06.174 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-17T09:37:06.174 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-17T09:37:06.174 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.6 2024-09-17T09:37:06.438 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:06 smithi103 systemd[1]: Stopping Ceph osd.6 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:37:06.734 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:06.437+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:37:06.734 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:06.437+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:37:06.734 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:06.437+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:38.132268+0000 front 2024-09-17T09:36:38.132331+0000 (oldest deadline 2024-09-17T09:37:03.932728+0000) 2024-09-17T09:37:06.734 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:06.495+0000 7f5b4b7e5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:37:06.734 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:06.495+0000 7f5b4b7e5640 -1 osd.6 143 *** Got signal Terminated *** 2024-09-17T09:37:06.735 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:06.495+0000 7f5b4b7e5640 -1 osd.6 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:37:07.005 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:06.732+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:07.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:06.732+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:07.006 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:06 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:06.732+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:07.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:07.473+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:37:07.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:07.473+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:37:07.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:07.473+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:38.132268+0000 front 2024-09-17T09:36:38.132331+0000 (oldest deadline 2024-09-17T09:37:03.932728+0000) 2024-09-17T09:37:07.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:07.722+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:07.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:07.722+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:07.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:07 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:07.722+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:08.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:08.427+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:37:08.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:08.427+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:37:08.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:08.427+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:38.132268+0000 front 2024-09-17T09:36:38.132331+0000 (oldest deadline 2024-09-17T09:37:03.932728+0000) 2024-09-17T09:37:09.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:08.761+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:09.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:08.761+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:09.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:08 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:08.761+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:09.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:09.425+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:37:09.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:09.425+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:37:09.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:09.425+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:38.132268+0000 front 2024-09-17T09:36:38.132331+0000 (oldest deadline 2024-09-17T09:37:03.932728+0000) 2024-09-17T09:37:10.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:09.805+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:10.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:09.805+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:10.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:09 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:09.805+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:10.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:10.427+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:37:10.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:10.427+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:37:10.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:10.427+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:38.132268+0000 front 2024-09-17T09:36:38.132331+0000 (oldest deadline 2024-09-17T09:37:03.932728+0000) 2024-09-17T09:37:11.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:10.795+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:11.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:10.795+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:11.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:10.795+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:11.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:10 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:10.795+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6830 osd.3 since back 2024-09-17T09:36:47.203526+0000 front 2024-09-17T09:36:47.203504+0000 (oldest deadline 2024-09-17T09:37:10.103494+0000) 2024-09-17T09:37:11.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:11.445+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:23.029971+0000 front 2024-09-17T09:36:23.030182+0000 (oldest deadline 2024-09-17T09:36:48.329806+0000) 2024-09-17T09:37:11.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:11.445+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.030867+0000 front 2024-09-17T09:36:30.030913+0000 (oldest deadline 2024-09-17T09:36:52.930757+0000) 2024-09-17T09:37:11.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6[58231]: 2024-09-17T09:37:11.445+0000 7f5b43df1640 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:38.132268+0000 front 2024-09-17T09:36:38.132331+0000 (oldest deadline 2024-09-17T09:37:03.932728+0000) 2024-09-17T09:37:12.215 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:11 smithi103 podman[80714]: 2024-09-17 09:37:11.949855773 +0000 UTC m=+5.559208886 container died a5355c784b3cbd4793b7f4be63ac7313236a448c44c44c14c6939336919d5877 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=reef-19e2a2b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True) 2024-09-17T09:37:12.215 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:11.840+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:12.215 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:11.840+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:12.215 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:11.840+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:12.215 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:11 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:11.840+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6830 osd.3 since back 2024-09-17T09:36:47.203526+0000 front 2024-09-17T09:36:47.203504+0000 (oldest deadline 2024-09-17T09:37:10.103494+0000) 2024-09-17T09:37:12.467 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:12 smithi103 podman[80714]: 2024-09-17 09:37:12.453887037 +0000 UTC m=+6.063240194 container cleanup a5355c784b3cbd4793b7f4be63ac7313236a448c44c44c14c6939336919d5877 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, RELEASE=reef-19e2a2b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-17T09:37:12.467 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:12 smithi103 bash[80714]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6 2024-09-17T09:37:13.005 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:12 smithi103 podman[80731]: 2024-09-17 09:37:12.697201147 +0000 UTC m=+0.740684346 container remove a5355c784b3cbd4793b7f4be63ac7313236a448c44c44c14c6939336919d5877 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-19e2a2b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-17T09:37:13.005 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:12.834+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:13.005 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:12.834+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:13.005 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:12.834+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:13.005 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:12 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:12.834+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6830 osd.3 since back 2024-09-17T09:36:47.203526+0000 front 2024-09-17T09:36:47.203504+0000 (oldest deadline 2024-09-17T09:37:10.103494+0000) 2024-09-17T09:37:13.256 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:13 smithi103 podman[80820]: 2024-09-17 09:37:12.907094995 +0000 UTC m=+0.016062525 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:37:13.256 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:13 smithi103 podman[80820]: 2024-09-17 09:37:13.014775717 +0000 UTC m=+0.123743278 container create 9e4e611bcbf8fd22a7ed1cd9c96bbbbcd7ed66af08a56bf3fafeb4d2e3e8bbf6 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-17T09:37:13.256 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:13 smithi103 podman[80820]: 2024-09-17 09:37:13.25031723 +0000 UTC m=+0.359284752 container init 9e4e611bcbf8fd22a7ed1cd9c96bbbbcd7ed66af08a56bf3fafeb4d2e3e8bbf6 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=reef-19e2a2b, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T09:37:13.256 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:13 smithi103 podman[80820]: 2024-09-17 09:37:13.253619027 +0000 UTC m=+0.362586543 container start 9e4e611bcbf8fd22a7ed1cd9c96bbbbcd7ed66af08a56bf3fafeb4d2e3e8bbf6 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=reef-19e2a2b, io.buildah.version=1.37.2) 2024-09-17T09:37:13.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:13 smithi103 podman[80820]: 2024-09-17 09:37:13.358791157 +0000 UTC m=+0.467758685 container attach 9e4e611bcbf8fd22a7ed1cd9c96bbbbcd7ed66af08a56bf3fafeb4d2e3e8bbf6 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-19e2a2b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:37:13.756 INFO:journalctl@ceph.osd.6.smithi103.stdout:Sep 17 09:37:13 smithi103 podman[80820]: 2024-09-17 09:37:13.435862635 +0000 UTC m=+0.544830169 container died 9e4e611bcbf8fd22a7ed1cd9c96bbbbcd7ed66af08a56bf3fafeb4d2e3e8bbf6 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-6-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=reef-19e2a2b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:37:13.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:37:13.516Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=9 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused" 2024-09-17T09:37:13.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:37:13.517Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused" 2024-09-17T09:37:13.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:13 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:37:13.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused" 2024-09-17T09:37:14.046 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:13.880+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:14.047 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:13.880+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:14.047 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:13.880+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:14.047 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:13 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:13.880+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6830 osd.3 since back 2024-09-17T09:36:47.203526+0000 front 2024-09-17T09:36:47.203504+0000 (oldest deadline 2024-09-17T09:37:10.103494+0000) 2024-09-17T09:37:14.073 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.6.service' 2024-09-17T09:37:14.113 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:37:14.113 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-17T09:37:14.113 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-17T09:37:14.113 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.7 2024-09-17T09:37:14.441 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:14 smithi103 systemd[1]: Stopping Ceph osd.7 for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:37:14.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:14.440+0000 7f86e1b21640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T09:37:14.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:14.440+0000 7f86e1b21640 -1 osd.7 143 *** Got signal Terminated *** 2024-09-17T09:37:14.756 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:14.440+0000 7f86e1b21640 -1 osd.7 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T09:37:15.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:14.853+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:15.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:14.853+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:15.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:14.853+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:15.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:14 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:14.853+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6830 osd.3 since back 2024-09-17T09:36:47.203526+0000 front 2024-09-17T09:36:47.203504+0000 (oldest deadline 2024-09-17T09:37:10.103494+0000) 2024-09-17T09:37:16.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:15.865+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:16.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:15.865+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:16.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:15.865+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:16.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:15 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:15.865+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6830 osd.3 since back 2024-09-17T09:36:47.203526+0000 front 2024-09-17T09:36:47.203504+0000 (oldest deadline 2024-09-17T09:37:10.103494+0000) 2024-09-17T09:37:17.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:16.884+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:17.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:16.884+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:17.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:16.884+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:17.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:16 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:16.884+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6830 osd.3 since back 2024-09-17T09:36:47.203526+0000 front 2024-09-17T09:36:47.203504+0000 (oldest deadline 2024-09-17T09:37:10.103494+0000) 2024-09-17T09:37:18.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:17.924+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:18.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:17.924+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:18.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:17.924+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:18.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:17 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:17.924+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6830 osd.3 since back 2024-09-17T09:36:47.203526+0000 front 2024-09-17T09:36:47.203504+0000 (oldest deadline 2024-09-17T09:37:10.103494+0000) 2024-09-17T09:37:19.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:18.927+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6806 osd.0 since back 2024-09-17T09:36:19.199625+0000 front 2024-09-17T09:36:19.199778+0000 (oldest deadline 2024-09-17T09:36:44.499630+0000) 2024-09-17T09:37:19.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:18.927+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6814 osd.1 since back 2024-09-17T09:36:30.301272+0000 front 2024-09-17T09:36:30.301287+0000 (oldest deadline 2024-09-17T09:36:55.601094+0000) 2024-09-17T09:37:19.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:18.927+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6822 osd.2 since back 2024-09-17T09:36:37.902078+0000 front 2024-09-17T09:36:37.902025+0000 (oldest deadline 2024-09-17T09:37:00.801925+0000) 2024-09-17T09:37:19.256 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:18 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7[60558]: 2024-09-17T09:37:18.927+0000 7f86de135640 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.29:6830 osd.3 since back 2024-09-17T09:36:47.203526+0000 front 2024-09-17T09:36:47.203504+0000 (oldest deadline 2024-09-17T09:37:10.103494+0000) 2024-09-17T09:37:20.093 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:19 smithi103 podman[80923]: 2024-09-17 09:37:19.814305423 +0000 UTC m=+5.480538695 container died 02f285335edfd31748e735f331423532a5bbadd35b3664fa7c05abd9dade99f6 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:37:20.475 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:20 smithi103 podman[80923]: 2024-09-17 09:37:20.181557075 +0000 UTC m=+5.847790336 container cleanup 02f285335edfd31748e735f331423532a5bbadd35b3664fa7c05abd9dade99f6 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7, RELEASE=reef-19e2a2b, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-17T09:37:20.475 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:20 smithi103 bash[80923]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7 2024-09-17T09:37:20.755 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:20 smithi103 podman[80950]: 2024-09-17 09:37:20.474548214 +0000 UTC m=+0.660439885 container remove 02f285335edfd31748e735f331423532a5bbadd35b3664fa7c05abd9dade99f6 (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-17T09:37:21.104 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:20 smithi103 podman[81041]: 2024-09-17 09:37:20.703596845 +0000 UTC m=+0.015868736 image pull 980a0bfcd684674f25c08d857ad8de3b1c8ae738870529edbf9aa463832c8d89 quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-17T09:37:21.104 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:20 smithi103 podman[81041]: 2024-09-17 09:37:20.808865287 +0000 UTC m=+0.121137177 container create b2f48fa7bf5dc11e014e426cbf287c3e4c066084898afa503ac82bb5ec69bb5a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T09:37:21.104 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:21 smithi103 podman[81041]: 2024-09-17 09:37:21.019579157 +0000 UTC m=+0.331851054 container init b2f48fa7bf5dc11e014e426cbf287c3e4c066084898afa503ac82bb5ec69bb5a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T09:37:21.104 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:21 smithi103 podman[81041]: 2024-09-17 09:37:21.023075521 +0000 UTC m=+0.335347420 container start b2f48fa7bf5dc11e014e426cbf287c3e4c066084898afa503ac82bb5ec69bb5a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=reef-19e2a2b, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T09:37:21.104 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:21 smithi103 podman[81041]: 2024-09-17 09:37:21.102772894 +0000 UTC m=+0.415044793 container attach b2f48fa7bf5dc11e014e426cbf287c3e4c066084898afa503ac82bb5ec69bb5a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, RELEASE=reef-19e2a2b) 2024-09-17T09:37:21.499 INFO:journalctl@ceph.osd.7.smithi103.stdout:Sep 17 09:37:21 smithi103 podman[81041]: 2024-09-17 09:37:21.210305293 +0000 UTC m=+0.522577190 container died b2f48fa7bf5dc11e014e426cbf287c3e4c066084898afa503ac82bb5ec69bb5a (image=quay.ceph.io/ceph-ci/ceph:19e2a2beb6f41a3ab62650e38614eb29b59dcbb9, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-osd-7-deactivate, ceph=True, io.buildah.version=1.37.2, RELEASE=reef-19e2a2b, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T09:37:21.809 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@osd.7.service' 2024-09-17T09:37:21.847 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:37:21.848 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-17T09:37:21.848 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-17T09:37:21.848 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a 2024-09-17T09:37:22.169 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:21 smithi103 systemd[1]: Stopping Ceph prometheus.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:37:22.453 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:37:22.167Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T09:37:22.453 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:37:22.167Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-17T09:37:22.453 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:37:22.167Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-17T09:37:22.453 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:37:22.167Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T09:37:22.454 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:37:22.167Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-17T09:37:22.454 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:37:22.168Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-17T09:37:22.454 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:37:22.168Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T09:37:22.454 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:37:22.168Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-17T09:37:22.454 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:37:22.168Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-17T09:37:22.454 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:37:22.168Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T09:37:22.454 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:37:22.168Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-17T09:37:22.454 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a[72071]: ts=2024-09-17T09:37:22.168Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-17T09:37:22.454 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 podman[81143]: 2024-09-17 09:37:22.177978226 +0000 UTC m=+0.104575370 container died f332a13317762305bea7f335cdc9f6972bebee8c65f6e2778d183a4fa12a885f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T09:37:22.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 podman[81143]: 2024-09-17 09:37:22.542197792 +0000 UTC m=+0.468794928 container cleanup f332a13317762305bea7f335cdc9f6972bebee8c65f6e2778d183a4fa12a885f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T09:37:22.756 INFO:journalctl@ceph.prometheus.a.smithi103.stdout:Sep 17 09:37:22 smithi103 bash[81143]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T09:37:22.963 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69135c5a-74d5-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-17T09:37:22.996 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T09:37:22.996 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-17T09:37:22.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T09:37:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] 2024-09-17T09:37:23.163 INFO:teuthology.orchestra.run.smithi029.stderr: [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] 2024-09-17T09:37:23.163 INFO:teuthology.orchestra.run.smithi029.stderr: [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] 2024-09-17T09:37:23.163 INFO:teuthology.orchestra.run.smithi029.stderr: [--retry RETRY] [--env ENV] [--no-container-init] 2024-09-17T09:37:23.164 INFO:teuthology.orchestra.run.smithi029.stderr: {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,exporter,host-maintenance,verify-prereqs} 2024-09-17T09:37:23.164 INFO:teuthology.orchestra.run.smithi029.stderr: ... 2024-09-17T09:37:23.164 INFO:teuthology.orchestra.run.smithi029.stderr:cephadm: error: unrecognized arguments: --keep-logs 2024-09-17T09:37:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-17T09:37:23.178 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T09:37:23.322 INFO:teuthology.orchestra.run.smithi103.stderr:usage: cephadm [-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR] 2024-09-17T09:37:23.322 INFO:teuthology.orchestra.run.smithi103.stderr: [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] 2024-09-17T09:37:23.322 INFO:teuthology.orchestra.run.smithi103.stderr: [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] 2024-09-17T09:37:23.323 INFO:teuthology.orchestra.run.smithi103.stderr: [--retry RETRY] [--env ENV] [--no-container-init] 2024-09-17T09:37:23.323 INFO:teuthology.orchestra.run.smithi103.stderr: {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,exporter,host-maintenance,verify-prereqs} 2024-09-17T09:37:23.323 INFO:teuthology.orchestra.run.smithi103.stderr: ... 2024-09-17T09:37:23.323 INFO:teuthology.orchestra.run.smithi103.stderr:cephadm: error: unrecognized arguments: --keep-logs 2024-09-17T09:37:23.336 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-17T09:37:23.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:37:23.363 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T09:37:23.388 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-17T09:37:23.390 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908288/remote/smithi029/crash 2024-09-17T09:37:23.391 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/crash -- . 2024-09-17T09:37:23.449 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908288/remote/smithi103/crash 2024-09-17T09:37:23.450 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/crash -- . 2024-09-17T09:37:23.494 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-17T09:37:23.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v CEPHADM_AGENT_DOWN | head -n 1 2024-09-17T09:37:23.521 INFO:teuthology.orchestra.run.smithi029.stderr:grep: /var/log/ceph/69135c5a-74d5-11ef-bceb-c7b262605968/ceph.log: No such file or directory 2024-09-17T09:37:23.523 INFO:tasks.cephadm:Compressing logs... 2024-09-17T09:37:23.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T09:37:23.566 DEBUG:teuthology.orchestra.run.smithi103:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T09:37:23.589 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-17T09:37:23.590 INFO:teuthology.orchestra.run.smithi103.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-17T09:37:23.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:37:23.517Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=9 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused" 2024-09-17T09:37:23.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:37:23.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused" 2024-09-17T09:37:23.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:23 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:37:23.519Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused" 2024-09-17T09:37:31.019 INFO:tasks.cephadm:Archiving logs... 2024-09-17T09:37:31.019 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908288/remote/smithi029/log 2024-09-17T09:37:31.021 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T09:37:31.916 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908288/remote/smithi103/log 2024-09-17T09:37:31.917 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T09:37:32.665 INFO:tasks.cephadm:Removing cluster... 2024-09-17T09:37:32.665 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 --force 2024-09-17T09:37:33.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:37:33.518Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=9 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused" 2024-09-17T09:37:33.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:37:33.519Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi029.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.29:8443: connect: connection refused" 2024-09-17T09:37:33.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:33 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:37:33.519Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi103.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.103:8443: connect: connection refused" 2024-09-17T09:37:34.613 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:37:34 smithi029 systemd[1]: Stopping Ceph node-exporter.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:37:34.870 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:37:34 smithi029 podman[99658]: 2024-09-17 09:37:34.612378512 +0000 UTC m=+0.116694708 container died 5f8842670a3c5e1e6e850c7abbe8958f2b6788df2b3f24cf28362ab8f79b9e03 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T09:37:35.232 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:37:34 smithi029 podman[99658]: 2024-09-17 09:37:34.969710791 +0000 UTC m=+0.474026986 container cleanup 5f8842670a3c5e1e6e850c7abbe8958f2b6788df2b3f24cf28362ab8f79b9e03 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T09:37:35.233 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:37:34 smithi029 bash[99658]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a 2024-09-17T09:37:35.548 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:37:35 smithi029 podman[99670]: 2024-09-17 09:37:35.23179566 +0000 UTC m=+0.618110106 container remove 5f8842670a3c5e1e6e850c7abbe8958f2b6788df2b3f24cf28362ab8f79b9e03 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T09:37:35.548 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:37:35 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:37:35.548 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:37:35 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-17T09:37:35.548 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:37:35 smithi029 systemd[1]: Stopped Ceph node-exporter.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:37:35.548 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Sep 17 09:37:35 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 2.159s CPU time. 2024-09-17T09:37:35.879 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:35 smithi029 systemd[1]: Stopping Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:37:35.879 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:35 smithi029 ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a[92448]: ts=2024-09-17T09:37:35.827Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-17T09:37:36.139 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:35 smithi029 podman[99777]: 2024-09-17 09:37:35.878530744 +0000 UTC m=+0.141595014 container died e3a985039ca09301d543ecc67f157d13e82e1971150d5457008f00284a0f2600 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:37:36.501 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:36 smithi029 podman[99777]: 2024-09-17 09:37:36.256409807 +0000 UTC m=+0.519474076 container cleanup e3a985039ca09301d543ecc67f157d13e82e1971150d5457008f00284a0f2600 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:37:36.501 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:36 smithi029 bash[99777]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a 2024-09-17T09:37:36.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:36 smithi029 podman[99789]: 2024-09-17 09:37:36.500160957 +0000 UTC m=+0.617490722 container remove e3a985039ca09301d543ecc67f157d13e82e1971150d5457008f00284a0f2600 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T09:37:36.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:36 smithi029 podman[99789]: 2024-09-17 09:37:36.584274209 +0000 UTC m=+0.701603981 volume remove 0a11199cf45b9d2b2bf735f5a176fbd172612b5ad73abaa1d3f071e0d4a04e39 2024-09-17T09:37:36.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:36 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T09:37:36.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:36 smithi029 systemd[1]: Stopped Ceph alertmanager.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:37:36.798 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Sep 17 09:37:36 smithi029 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.059s CPU time. 2024-09-17T09:38:11.914 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 69135c5a-74d5-11ef-bceb-c7b262605968 --force 2024-09-17T09:38:13.732 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:38:13 smithi103 systemd[1]: Stopping Ceph node-exporter.b for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:38:14.006 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:38:13 smithi103 podman[81646]: 2024-09-17 09:38:13.888912993 +0000 UTC m=+0.213852939 container died 1f9c124dcc9ad4f920e6de8fbb3e6e52a4971296508f0e99141a776f70e15b07 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T09:38:14.480 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:38:14 smithi103 podman[81646]: 2024-09-17 09:38:14.311409855 +0000 UTC m=+0.636349787 container cleanup 1f9c124dcc9ad4f920e6de8fbb3e6e52a4971296508f0e99141a776f70e15b07 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T09:38:14.480 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:38:14 smithi103 bash[81646]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b 2024-09-17T09:38:14.756 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:38:14 smithi103 podman[81659]: 2024-09-17 09:38:14.479284441 +0000 UTC m=+0.590072770 container remove 1f9c124dcc9ad4f920e6de8fbb3e6e52a4971296508f0e99141a776f70e15b07 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T09:38:14.756 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:38:14 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:38:14.756 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:38:14 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-17T09:38:14.756 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:38:14 smithi103 systemd[1]: Stopped Ceph node-exporter.b for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:38:14.756 INFO:journalctl@ceph.node-exporter.b.smithi103.stdout:Sep 17 09:38:14 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 2.042s CPU time. 2024-09-17T09:38:15.135 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:38:14 smithi103 systemd[1]: Stopping Ceph grafana.a for 69135c5a-74d5-11ef-bceb-c7b262605968... 2024-09-17T09:38:15.424 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:38:15 smithi103 podman[81772]: 2024-09-17 09:38:15.134450816 +0000 UTC m=+0.122988764 container died 0053793acd878aa815380e7a6168461856eb3bf46a698426a9e4816059dca462 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, description=Ceph Grafana Container, vcs-type=git, build-date=2022-04-29T11:48:11.055897) 2024-09-17T09:38:15.425 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:38:15 smithi103 podman[81772]: 2024-09-17 09:38:15.423741754 +0000 UTC m=+0.412279699 container cleanup 0053793acd878aa815380e7a6168461856eb3bf46a698426a9e4816059dca462 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, vcs-type=git, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.openshift.expose-services=, description=Ceph Grafana Container) 2024-09-17T09:38:15.756 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:38:15 smithi103 bash[81772]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a 2024-09-17T09:38:15.756 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:38:15 smithi103 podman[81784]: 2024-09-17 09:38:15.591783746 +0000 UTC m=+0.459755325 container remove 0053793acd878aa815380e7a6168461856eb3bf46a698426a9e4816059dca462 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-69135c5a-74d5-11ef-bceb-c7b262605968-grafana-a, vendor=Red Hat, Inc., version=8.5, name=ubi8, description=Ceph Grafana Container, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, release=239.1651231664) 2024-09-17T09:38:15.756 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:38:15 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T09:38:16.238 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:38:15 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-17T09:38:16.238 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:38:15 smithi103 systemd[1]: Stopped Ceph grafana.a for 69135c5a-74d5-11ef-bceb-c7b262605968. 2024-09-17T09:38:16.238 INFO:journalctl@ceph.grafana.a.smithi103.stdout:Sep 17 09:38:15 smithi103 systemd[1]: ceph-69135c5a-74d5-11ef-bceb-c7b262605968@grafana.a.service: Consumed 4.139s CPU time. 2024-09-17T09:38:38.910 INFO:tasks.cephadm:Removing cephadm ... 2024-09-17T09:38:38.910 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T09:38:38.932 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T09:38:38.949 INFO:tasks.cephadm:Teardown complete 2024-09-17T09:38:38.949 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T09:38:38.960 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T09:38:38.960 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T09:38:38.975 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T09:38:38.990 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-09-17T09:38:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T09:38:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-09-17T09:38:38.994 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv01.front.sepia.ceph.com 3 6 377 12 -315us[ -302us] +/- 47ms 2024-09-17T09:38:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 10 -724us[ -724us] +/- 67ms 2024-09-17T09:38:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +1723us[+1723us] +/- 88ms 2024-09-17T09:38:38.995 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T09:38:39.003 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-09-17T09:38:39.007 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T09:38:39.007 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-09-17T09:38:39.007 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv01.front.sepia.ceph.com 3 6 377 15 -300us[ -313us] +/- 47ms 2024-09-17T09:38:39.008 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 11 -699us[ -699us] +/- 67ms 2024-09-17T09:38:39.008 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +1721us[+1721us] +/- 87ms 2024-09-17T09:38:39.008 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T09:38:39.009 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T09:38:39.022 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T09:38:39.023 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T09:38:39.067 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T09:38:39.152 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T09:38:39.226 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T09:38:39.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T09:38:39.259 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T09:38:39.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T09:38:39.289 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T09:38:39.298 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T09:38:39.306 INFO:teuthology.task.internal:Duration was 1611.713018 seconds 2024-09-17T09:38:39.306 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T09:38:39.315 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T09:38:39.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T09:38:39.317 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T09:38:39.357 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T09:38:39.369 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T09:38:39.840 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T09:38:39.840 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-09-17T09:38:39.841 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T09:38:39.906 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-09-17T09:38:39.906 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T09:38:39.934 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T09:38:39.934 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T09:38:39.949 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T09:38:40.002 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T09:38:40.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T09:38:40.021 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T09:38:40.551 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T09:38:40.560 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T09:38:40.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T09:38:40.586 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T09:38:40.611 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T09:38:40.623 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T09:38:40.629 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T09:38:40.651 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-09-17T09:38:40.674 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-09-17T09:38:40.699 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T09:38:40.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T09:38:40.729 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T09:38:40.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T09:38:40.754 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T09:38:40.763 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T09:38:40.764 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908288/remote/smithi029 2024-09-17T09:38:40.764 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T09:38:40.816 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-16_22:08:02-orch-reef-distro-default-smithi/7908288/remote/smithi103 2024-09-17T09:38:40.816 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T09:38:40.858 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T09:38:40.858 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T09:38:40.860 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T09:38:40.906 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T09:38:40.917 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T09:38:40.917 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T09:38:40.926 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T09:38:40.927 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T09:38:40.929 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T09:38:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 09:38 /home/ubuntu/cephtest 2024-09-17T09:38:40.962 INFO:teuthology.orchestra.run.smithi103.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 09:38 /home/ubuntu/cephtest 2024-09-17T09:38:40.963 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T09:38:40.972 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T09:38:40.996 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1611.7130184173584 owner: scheduled_teuthology@teuthology success: true 2024-09-17T09:38:40.996 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T09:38:41.091 INFO:teuthology.run:pass