2024-09-20T22:20:33.919 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-20T22:20:33.935 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-20T22:20:34.004 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913005 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7913005' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - CEPHADM_AGENT_DOWN log-only-match: - CEPHADM_ sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 workunit: branch: squid sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 9344 sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 sleep_before_teardown: 0 subset: 56/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 targets: smithi115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/RBWstXqTri8ixj6qGTTVnfjuyGgC4jluW0XiigOACYwrU+/Xyp4bCpc0nBlko8Inj1Wk6jvOm/UkcQt1LejM= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHqefG9stkpQIKCteZazs+FpWIIwOcFw2p5/zNOQSRGoM71JHaR4HNqUmKaWwawaF59+k09cbgHNeTKgWLb74G8= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-20T22:20:34.004 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa; will attempt to use it 2024-09-20T22:20:34.005 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks 2024-09-20T22:20:34.005 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-20T22:20:34.007 INFO:teuthology.task.internal:Checking packages... 2024-09-20T22:20:34.025 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9e2ac70bb7aea67d34f0d50002344abbe7803736' 2024-09-20T22:20:34.025 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T22:20:34.025 INFO:teuthology.packaging:ref: None 2024-09-20T22:20:34.025 INFO:teuthology.packaging:tag: None 2024-09-20T22:20:34.026 INFO:teuthology.packaging:branch: squid 2024-09-20T22:20:34.026 INFO:teuthology.packaging:sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:20:34.026 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-20T22:20:34.235 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-373.g4e45e6b4 2024-09-20T22:20:34.236 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-20T22:20:34.250 INFO:teuthology.task.internal:no buildpackages task found 2024-09-20T22:20:34.250 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-20T22:20:34.256 INFO:teuthology.task.internal:Saving configuration 2024-09-20T22:20:34.267 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-20T22:20:34.274 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-20T22:20:34.301 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913005', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 22:15:16.462984', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/RBWstXqTri8ixj6qGTTVnfjuyGgC4jluW0XiigOACYwrU+/Xyp4bCpc0nBlko8Inj1Wk6jvOm/UkcQt1LejM='} 2024-09-20T22:20:34.324 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913005', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 22:15:16.461682', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHqefG9stkpQIKCteZazs+FpWIIwOcFw2p5/zNOQSRGoM71JHaR4HNqUmKaWwawaF59+k09cbgHNeTKgWLb74G8='} 2024-09-20T22:20:34.325 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-20T22:20:34.331 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-20T22:20:34.331 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-20T22:20:34.331 INFO:teuthology.run_tasks:Running task console_log... 2024-09-20T22:20:34.388 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4ae8970160>, signals=[15]) 2024-09-20T22:20:34.388 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-20T22:20:34.394 INFO:teuthology.task.internal:Opening connections... 2024-09-20T22:20:34.394 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2024-09-20T22:20:34.395 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:20:34.471 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-09-20T22:20:34.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:20:34.543 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-20T22:20:34.550 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2024-09-20T22:20:34.568 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2024-09-20T22:20:34.568 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2024-09-20T22:20:34.625 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream" 2024-09-20T22:20:34.625 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="9" 2024-09-20T22:20:34.625 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos" 2024-09-20T22:20:34.625 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora" 2024-09-20T22:20:34.625 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="9" 2024-09-20T22:20:34.625 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el9" 2024-09-20T22:20:34.626 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T22:20:34.626 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2024-09-20T22:20:34.626 INFO:teuthology.orchestra.run.smithi115.stdout:LOGO="fedora-logo-icon" 2024-09-20T22:20:34.626 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T22:20:34.626 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/" 2024-09-20T22:20:34.626 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T22:20:34.626 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T22:20:34.626 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T22:20:34.627 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2024-09-20T22:20:34.650 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-09-20T22:20:34.669 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-09-20T22:20:34.669 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-09-20T22:20:34.727 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-09-20T22:20:34.727 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-09-20T22:20:34.727 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-09-20T22:20:34.727 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-09-20T22:20:34.727 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-09-20T22:20:34.727 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-09-20T22:20:34.727 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T22:20:34.727 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-09-20T22:20:34.727 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-09-20T22:20:34.727 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T22:20:34.727 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-09-20T22:20:34.727 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T22:20:34.728 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T22:20:34.728 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T22:20:34.728 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-09-20T22:20:34.750 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-20T22:20:34.759 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-20T22:20:34.764 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-20T22:20:34.764 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2024-09-20T22:20:34.767 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-09-20T22:20:34.784 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-20T22:20:34.790 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-20T22:20:34.790 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2024-09-20T22:20:34.824 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-09-20T22:20:34.944 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-20T22:20:34.994 INFO:teuthology.run_tasks:Running task kernel... 2024-09-20T22:20:35.008 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-20T22:20:35.008 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-20T22:20:35.008 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T22:20:35.009 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T22:20:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:20:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:20:35.026 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-09-20T22:20:35.026 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-09-20T22:20:35.083 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-503.el9.x86_64 2024-09-20T22:20:35.083 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-503.el9.x86_64 2024-09-20T22:20:35.083 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-09-20T22:20:35.084 INFO:teuthology.orchestra.run.smithi115.stdout:5.14.0-503.el9.x86_64 2024-09-20T22:20:35.084 INFO:teuthology.task.kernel:Running kernel on smithi115: 5.14.0-503.el9.x86_64 2024-09-20T22:20:35.084 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum install -y kernel 2024-09-20T22:20:37.546 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-20T22:20:37.800 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 8.6 MB/s | 8.3 MB 00:00 2024-09-20T22:20:39.322 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-20T22:20:39.709 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-20T22:20:43.831 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.5 MB 00:00 2024-09-20T22:20:43.885 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-20T22:20:45.511 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 19 kB 00:00 2024-09-20T22:20:45.640 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 19 kB 00:00 2024-09-20T22:20:46.573 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-20T22:20:46.619 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-20T22:20:51.987 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-20T22:20:52.141 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-20T22:20:53.885 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T22:20:53.885 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T22:20:53.985 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-20T22:20:53.995 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-20T22:20:53.995 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-09-20T22:20:53.995 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-20T22:20:53.995 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-20T22:20:53.995 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T22:20:53.995 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-20T22:20:53.995 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T22:20:53.995 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T22:20:53.995 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T22:20:53.995 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:20:53.996 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-20T22:20:53.996 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-20T22:20:53.996 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-09-20T22:20:53.996 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:20:53.996 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 79 M 2024-09-20T22:20:53.996 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 126 M 2024-09-20T22:20:53.996 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-20T22:20:54.038 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T22:20:54.038 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T22:20:54.128 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-09-20T22:20:54.132 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-20T22:20:54.132 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-09-20T22:20:54.132 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-20T22:20:54.132 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-09-20T22:20:54.133 INFO:teuthology.orchestra.run.smithi115.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T22:20:54.133 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-09-20T22:20:54.133 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T22:20:54.133 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T22:20:54.133 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T22:20:54.133 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:20:54.133 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-09-20T22:20:54.133 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-09-20T22:20:54.133 INFO:teuthology.orchestra.run.smithi115.stdout:Install 4 Packages 2024-09-20T22:20:54.133 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:20:54.134 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 79 M 2024-09-20T22:20:54.134 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 126 M 2024-09-20T22:20:54.134 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-09-20T22:20:54.179 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 271 kB/s | 36 kB 00:00 2024-09-20T22:20:54.317 INFO:teuthology.orchestra.run.smithi115.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 265 kB/s | 36 kB 00:00 2024-09-20T22:20:54.663 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-09-20T22:20:55.130 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 32 MB/s | 35 MB 00:01 2024-09-20T22:20:55.389 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 24 MB/s | 29 MB 00:01 2024-09-20T22:20:55.389 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:20:55.389 INFO:teuthology.orchestra.run.smithi153.stdout:Total 57 MB/s | 79 MB 00:01 2024-09-20T22:20:55.900 INFO:teuthology.orchestra.run.smithi115.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 9.1 MB/s | 16 MB 00:01 2024-09-20T22:20:56.077 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-20T22:20:56.437 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-20T22:20:56.437 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-20T22:20:57.374 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-20T22:20:57.374 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-20T22:20:58.508 INFO:teuthology.orchestra.run.smithi115.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 6.8 MB/s | 29 MB 00:04 2024-09-20T22:20:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 7.2 MB/s | 35 MB 00:04 2024-09-20T22:20:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:20:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:Total 16 MB/s | 79 MB 00:04 2024-09-20T22:20:59.667 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-09-20T22:20:59.817 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-20T22:21:00.022 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-09-20T22:21:00.022 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-09-20T22:21:00.510 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T22:21:00.591 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T22:21:00.998 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-09-20T22:21:00.998 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-09-20T22:21:03.353 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T22:21:03.408 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-09-20T22:21:03.485 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T22:21:04.060 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T22:21:04.142 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T22:21:06.887 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T22:21:07.012 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T22:21:08.324 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T22:21:08.360 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:21:11.852 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T22:21:11.954 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:21:13.203 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:21:16.827 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:21:51.449 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:21:51.490 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:21:51.885 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:21:51.885 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T22:21:51.886 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T22:21:51.886 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T22:21:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:21:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:21:52.788 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-20T22:21:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T22:21:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T22:21:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T22:21:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T22:21:52.788 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:21:52.788 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-20T22:21:53.159 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T22:21:53.247 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T22:21:53.247 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T22:21:53.247 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T22:21:53.247 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T22:21:53.247 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T22:21:53.247 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-20T22:21:53.247 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T22:21:53.247 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T22:21:53.305 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T22:21:53.306 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T22:21:53.306 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-09-20T22:21:53.338 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-09-20T22:21:53.338 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-09-20T22:21:53.338 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-09-20T22:21:53.338 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 82.el9 2024-09-20T22:21:53.338 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-09-20T22:21:53.338 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T22:21:53.338 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-09-20T22:21:53.338 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8274670 2024-09-20T22:21:53.338 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-09-20T22:21:53.339 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:21:53.340 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T22:21:53.340 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T22:21:53.340 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-508.el9.x86_64 2024-09-20T22:21:53.341 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T22:21:54.031 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-09-20T22:21:55.139 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:21:55.178 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:21:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T22:21:55.354 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-09-20T22:21:55.360 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-20T22:21:55.375 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.87SzKr56xj 2024-09-20T22:21:55.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.87SzKr56xj 2024-09-20T22:21:55.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.87SzKr56xj 2024-09-20T22:21:55.551 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:21:55.552 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T22:21:55.552 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T22:21:55.552 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T22:21:55.595 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.87SzKr56xj is 6KB 2024-09-20T22:21:55.643 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.87SzKr56xj 2024-09-20T22:21:55.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T22:21:55.730 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T22:21:55.730 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T22:21:55.730 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T22:21:55.730 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T22:21:55.731 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T22:21:55.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-09-20T22:21:56.386 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:21:56.387 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:21:56.387 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-09-20T22:21:56.387 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T22:21:56.387 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T22:21:56.387 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T22:21:56.387 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T22:21:56.387 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:21:56.387 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-09-20T22:21:56.728 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T22:21:56.850 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T22:21:56.850 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T22:21:56.850 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T22:21:56.850 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T22:21:56.850 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T22:21:56.851 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-20T22:21:56.851 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi115.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T22:21:56.851 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T22:21:56.908 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T22:21:56.909 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T22:21:56.909 DEBUG:teuthology.orchestra.run.smithi115:> sudo rpm -qi grub2-tools 2024-09-20T22:21:56.943 INFO:teuthology.orchestra.run.smithi115.stdout:Name : grub2-tools 2024-09-20T22:21:56.943 INFO:teuthology.orchestra.run.smithi115.stdout:Epoch : 1 2024-09-20T22:21:56.943 INFO:teuthology.orchestra.run.smithi115.stdout:Version : 2.06 2024-09-20T22:21:56.943 INFO:teuthology.orchestra.run.smithi115.stdout:Release : 82.el9 2024-09-20T22:21:56.943 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: x86_64 2024-09-20T22:21:56.943 INFO:teuthology.orchestra.run.smithi115.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T22:21:56.943 INFO:teuthology.orchestra.run.smithi115.stdout:Group : Unspecified 2024-09-20T22:21:56.944 INFO:teuthology.orchestra.run.smithi115.stdout:Size : 8274670 2024-09-20T22:21:56.944 INFO:teuthology.orchestra.run.smithi115.stdout:License : GPLv3+ 2024-09-20T22:21:56.944 INFO:teuthology.orchestra.run.smithi115.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T22:21:56.944 INFO:teuthology.orchestra.run.smithi115.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T22:21:56.944 INFO:teuthology.orchestra.run.smithi115.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T22:21:56.944 INFO:teuthology.orchestra.run.smithi115.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T22:21:56.944 INFO:teuthology.orchestra.run.smithi115.stdout:Packager : builder@centos.org 2024-09-20T22:21:56.944 INFO:teuthology.orchestra.run.smithi115.stdout:Vendor : CentOS 2024-09-20T22:21:56.944 INFO:teuthology.orchestra.run.smithi115.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T22:21:56.944 INFO:teuthology.orchestra.run.smithi115.stdout:Summary : Support tools for GRUB. 2024-09-20T22:21:56.945 INFO:teuthology.orchestra.run.smithi115.stdout:Description : 2024-09-20T22:21:56.946 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:21:56.946 INFO:teuthology.orchestra.run.smithi115.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T22:21:56.946 INFO:teuthology.orchestra.run.smithi115.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T22:21:56.946 INFO:teuthology.orchestra.run.smithi115.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T22:21:56.946 INFO:teuthology.orchestra.run.smithi115.stdout:hardware devices. 2024-09-20T22:21:56.946 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:21:56.946 INFO:teuthology.orchestra.run.smithi115.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T22:21:56.947 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T22:21:56.947 INFO:teuthology.task.kernel:Updating grub on smithi115 to boot 5.14.0-508.el9.x86_64 2024-09-20T22:21:56.947 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T22:21:57.607 INFO:teuthology.orchestra.run.smithi115.stderr:Generating grub configuration file ... 2024-09-20T22:21:58.860 INFO:teuthology.orchestra.run.smithi115.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T22:21:58.891 INFO:teuthology.orchestra.run.smithi115.stderr:done 2024-09-20T22:21:58.893 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2024-09-20T22:21:58.908 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.yS9qteFlD6 2024-09-20T22:21:58.909 DEBUG:teuthology.orchestra.run.smithi115:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yS9qteFlD6 2024-09-20T22:21:58.977 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.yS9qteFlD6 2024-09-20T22:21:59.113 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.yS9qteFlD6 is 6KB 2024-09-20T22:21:59.161 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.yS9qteFlD6 2024-09-20T22:21:59.176 DEBUG:teuthology.orchestra.run.smithi115:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T22:21:59.238 INFO:teuthology.orchestra.run.smithi115.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T22:21:59.239 INFO:teuthology.orchestra.run.smithi115.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T22:21:59.239 INFO:teuthology.orchestra.run.smithi115.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T22:21:59.239 INFO:teuthology.orchestra.run.smithi115.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T22:21:59.240 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T22:21:59.383 DEBUG:teuthology.orchestra.run.smithi115:> sudo shutdown -r now 2024-09-20T22:22:25.907 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-20T22:22:25.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-20T22:22:25.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:22:29.388 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-20T22:22:29.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-20T22:22:29.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:22:44.463 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-09-20T22:22:47.791 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-09-20T22:22:53.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-20T22:22:53.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:22:56.527 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-09-20T22:22:56.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-20T22:22:56.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:22:59.855 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-09-20T22:23:08.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-20T22:23:08.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:23:11.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-20T22:23:11.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:23:30.287 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-09-20T22:23:45.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-20T22:23:45.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:23:45.710 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-09-20T22:23:46.089 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-20T22:23:46.089 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-20T22:23:46.090 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-09-20T22:23:46.104 INFO:teuthology.orchestra.run.smithi115.stdout:5.14.0-508.el9.x86_64 2024-09-20T22:23:46.104 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T22:23:46.104 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T22:23:46.105 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T22:23:47.106 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-20T22:23:47.106 DEBUG:teuthology.orchestra.run.smithi115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T22:23:47.194 INFO:teuthology.orchestra.run.smithi115.stdout:ttyS1 2024-09-20T22:23:47.224 DEBUG:teuthology.parallel:result is None 2024-09-20T22:24:08.534 DEBUG:teuthology.orchestra.remote:timed out 2024-09-20T22:24:23.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-20T22:24:23.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:24:23.824 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-09-20T22:24:24.194 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-20T22:24:24.194 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-20T22:24:24.194 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-09-20T22:24:24.210 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-508.el9.x86_64 2024-09-20T22:24:24.211 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T22:24:24.211 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T22:24:24.211 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T22:24:25.211 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-20T22:24:25.211 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T22:24:25.269 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-09-20T22:24:25.299 DEBUG:teuthology.parallel:result is None 2024-09-20T22:24:25.300 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-20T22:24:25.305 INFO:teuthology.task.internal:Creating test directory... 2024-09-20T22:24:25.305 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T22:24:25.308 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T22:24:25.326 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-20T22:24:25.342 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-20T22:24:25.366 INFO:teuthology.task.internal:Creating archive directory... 2024-09-20T22:24:25.366 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T22:24:25.369 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T22:24:25.401 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-20T22:24:25.407 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-20T22:24:25.407 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T22:24:25.429 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T22:24:25.464 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T22:24:25.476 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T22:24:25.489 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T22:24:25.501 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T22:24:25.503 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-20T22:24:25.510 INFO:teuthology.task.internal:Configuring sudo... 2024-09-20T22:24:25.510 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T22:24:25.520 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T22:24:25.547 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-20T22:24:25.556 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-20T22:24:25.556 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T22:24:25.588 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T22:24:25.605 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T22:24:25.687 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T22:24:25.736 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T22:24:25.819 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T22:24:25.861 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-20T22:24:25.861 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T22:24:25.926 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T22:24:25.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T22:24:26.037 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T22:24:26.081 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T22:24:26.123 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-20T22:24:26.123 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T22:24:26.190 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2024-09-20T22:24:26.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-09-20T22:24:26.233 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T22:24:26.277 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T22:24:26.810 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-20T22:24:26.818 INFO:teuthology.task.internal:Starting timer... 2024-09-20T22:24:26.818 INFO:teuthology.run_tasks:Running task pcp... 2024-09-20T22:24:26.845 INFO:teuthology.run_tasks:Running task selinux... 2024-09-20T22:24:26.853 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2024-09-20T22:24:26.912 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: 2024-09-20T22:24:26.913 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-09-20T22:24:26.956 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-09-20T22:24:26.958 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-20T22:24:26.958 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2024-09-20T22:24:26.984 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2024-09-20T22:24:26.984 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-09-20T22:24:27.027 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-09-20T22:24:27.027 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi115.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2024-09-20T22:24:27.027 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-20T22:24:27.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:24:27.057 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-20T22:24:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:24:27.096 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-20T22:24:27.096 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2024-09-20T22:24:27.134 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-09-20T22:24:27.171 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-20T22:24:27.180 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-20T22:24:27.190 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-20T22:24:27.191 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi115.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-20T22:28:13.147 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi115.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-09-20T22:28:13.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-20T22:28:13.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:28:13.224 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-09-20T22:28:13.301 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-09-20T22:28:13.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-20T22:28:13.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:28:13.374 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-09-20T22:28:13.454 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-20T22:28:13.454 INFO:teuthology.run_tasks:Running task clock... 2024-09-20T22:28:13.469 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-20T22:28:13.469 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T22:28:13.469 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T22:28:13.472 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T22:28:13.472 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T22:28:13.497 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T22:28:13.512 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T22:28:13.523 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T22:28:13.535 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2024-09-20T22:28:13.537 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T22:28:13.546 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2024-09-20T22:28:13.560 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T22:28:13.562 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-09-20T22:28:13.574 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-09-20T22:28:13.574 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T22:28:13.588 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T22:28:13.602 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T22:28:13.637 INFO:teuthology.orchestra.run.smithi115.stderr:bash: line 1: ntpq: command not found 2024-09-20T22:28:13.640 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T22:28:13.641 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-09-20T22:28:13.641 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:28:13.641 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:28:13.641 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:28:13.641 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:28:13.663 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-09-20T22:28:13.667 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T22:28:13.667 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-09-20T22:28:13.668 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:28:13.668 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:28:13.668 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:28:13.668 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:28:13.668 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-20T22:28:13.759 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'CEPHADM_AGENT_DOWN'], 'log-only-match': ['CEPHADM_'], 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736'} 2024-09-20T22:28:13.759 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-09-20T22:28:13.760 INFO:tasks.cephadm:Cluster fsid is 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:28:13.760 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-20T22:28:13.760 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.115', 'mon.c': '[v2:172.21.15.115:3301,v1:172.21.15.115:6790]', 'mon.b': '172.21.15.153'} 2024-09-20T22:28:13.760 INFO:tasks.cephadm:First mon is mon.a on smithi115 2024-09-20T22:28:13.760 INFO:tasks.cephadm:First mgr is y 2024-09-20T22:28:13.760 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-20T22:28:13.760 DEBUG:teuthology.orchestra.run.smithi115:> sudo hostname $(hostname -s) 2024-09-20T22:28:13.786 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-09-20T22:28:13.812 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-09-20T22:28:13.812 DEBUG:teuthology.orchestra.run.smithi115:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T22:28:13.997 INFO:teuthology.orchestra.run.smithi115.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Sep 20 22:28 /home/ubuntu/cephtest/cephadm 2024-09-20T22:28:13.997 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T22:28:14.121 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Sep 20 22:28 /home/ubuntu/cephtest/cephadm 2024-09-20T22:28:14.122 DEBUG:teuthology.orchestra.run.smithi115:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T22:28:14.138 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T22:28:14.163 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-09-20T22:28:14.164 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-09-20T22:28:14.180 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-09-20T22:28:15.736 INFO:teuthology.orchestra.run.smithi115.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-09-20T22:28:15.748 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-09-20T22:28:27.772 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-20T22:28:27.772 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-09-20T22:28:27.772 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-09-20T22:28:27.772 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-09-20T22:28:27.772 INFO:teuthology.orchestra.run.smithi153.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-09-20T22:28:27.772 INFO:teuthology.orchestra.run.smithi153.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-09-20T22:28:27.773 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-20T22:28:27.773 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-20T22:28:27.788 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:28:27.788 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-09-20T22:28:27.788 INFO:teuthology.orchestra.run.smithi115.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-09-20T22:28:27.788 INFO:teuthology.orchestra.run.smithi115.stdout: "repo_digests": [ 2024-09-20T22:28:27.788 INFO:teuthology.orchestra.run.smithi115.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-09-20T22:28:27.788 INFO:teuthology.orchestra.run.smithi115.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-09-20T22:28:27.788 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-09-20T22:28:27.788 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:28:27.806 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph 2024-09-20T22:28:27.864 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-09-20T22:28:27.891 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /etc/ceph 2024-09-20T22:28:27.915 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-09-20T22:28:27.956 INFO:tasks.cephadm:Writing seed config... 2024-09-20T22:28:27.956 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-20T22:28:27.957 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-20T22:28:27.957 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-20T22:28:27.957 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-09-20T22:28:27.957 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-20T22:28:27.957 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-20T22:28:27.957 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-20T22:28:27.957 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-20T22:28:27.957 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-20T22:28:27.957 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-20T22:28:27.958 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-20T22:28:27.973 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9fe0f662-779f-11ef-baf5-efdc52797490 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-20T22:28:27.973 DEBUG:teuthology.orchestra.run.smithi115:mon.a> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.a.service 2024-09-20T22:28:28.015 DEBUG:teuthology.orchestra.run.smithi115:mgr.y> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.y.service 2024-09-20T22:28:28.057 INFO:tasks.cephadm:Bootstrapping... 2024-09-20T22:28:28.058 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.115 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-20T22:28:28.244 INFO:teuthology.orchestra.run.smithi115.stderr:-------------------------------------------------------------------------------- 2024-09-20T22:28:28.244 INFO:teuthology.orchestra.run.smithi115.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '9fe0f662-779f-11ef-baf5-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.115', '--skip-admin-label'] 2024-09-20T22:28:28.266 INFO:teuthology.orchestra.run.smithi115.stderr:/bin/podman: 5.2.2 2024-09-20T22:28:28.269 INFO:teuthology.orchestra.run.smithi115.stderr:Verifying podman|docker is present... 2024-09-20T22:28:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:/bin/podman: 5.2.2 2024-09-20T22:28:28.291 INFO:teuthology.orchestra.run.smithi115.stderr:Verifying lvm2 is present... 2024-09-20T22:28:28.291 INFO:teuthology.orchestra.run.smithi115.stderr:Verifying time synchronization is in place... 2024-09-20T22:28:28.298 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T22:28:28.304 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2024-09-20T22:28:28.311 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: enabled 2024-09-20T22:28:28.317 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: active 2024-09-20T22:28:28.318 INFO:teuthology.orchestra.run.smithi115.stderr:Unit chronyd.service is enabled and running 2024-09-20T22:28:28.318 INFO:teuthology.orchestra.run.smithi115.stderr:Repeating the final host check... 2024-09-20T22:28:28.337 INFO:teuthology.orchestra.run.smithi115.stderr:/bin/podman: 5.2.2 2024-09-20T22:28:28.340 INFO:teuthology.orchestra.run.smithi115.stderr:podman (/bin/podman) version 5.2.2 is present 2024-09-20T22:28:28.340 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl is present 2024-09-20T22:28:28.340 INFO:teuthology.orchestra.run.smithi115.stderr:lvcreate is present 2024-09-20T22:28:28.346 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T22:28:28.352 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2024-09-20T22:28:28.359 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: enabled 2024-09-20T22:28:28.365 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: active 2024-09-20T22:28:28.366 INFO:teuthology.orchestra.run.smithi115.stderr:Unit chronyd.service is enabled and running 2024-09-20T22:28:28.366 INFO:teuthology.orchestra.run.smithi115.stderr:Host looks OK 2024-09-20T22:28:28.366 INFO:teuthology.orchestra.run.smithi115.stderr:Cluster fsid: 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:28:28.366 INFO:teuthology.orchestra.run.smithi115.stderr:Acquiring lock 140519728644736 on /run/cephadm/9fe0f662-779f-11ef-baf5-efdc52797490.lock 2024-09-20T22:28:28.366 INFO:teuthology.orchestra.run.smithi115.stderr:Lock 140519728644736 acquired on /run/cephadm/9fe0f662-779f-11ef-baf5-efdc52797490.lock 2024-09-20T22:28:28.367 INFO:teuthology.orchestra.run.smithi115.stderr:Verifying IP 172.21.15.115 port 3300 ... 2024-09-20T22:28:28.367 INFO:teuthology.orchestra.run.smithi115.stderr:Verifying IP 172.21.15.115 port 6789 ... 2024-09-20T22:28:28.367 INFO:teuthology.orchestra.run.smithi115.stderr:Base mon IP is 172.21.15.115, final addrv is [v2:172.21.15.115:3300,v1:172.21.15.115:6789] 2024-09-20T22:28:28.369 INFO:teuthology.orchestra.run.smithi115.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.115 metric 100 2024-09-20T22:28:28.369 INFO:teuthology.orchestra.run.smithi115.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.115 metric 100 2024-09-20T22:28:28.372 INFO:teuthology.orchestra.run.smithi115.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-09-20T22:28:28.372 INFO:teuthology.orchestra.run.smithi115.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-20T22:28:28.372 INFO:teuthology.orchestra.run.smithi115.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-20T22:28:28.374 INFO:teuthology.orchestra.run.smithi115.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-20T22:28:28.374 INFO:teuthology.orchestra.run.smithi115.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-09-20T22:28:28.375 INFO:teuthology.orchestra.run.smithi115.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-09-20T22:28:28.375 INFO:teuthology.orchestra.run.smithi115.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-20T22:28:28.375 INFO:teuthology.orchestra.run.smithi115.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:ce8f/64 scope link noprefixroute 2024-09-20T22:28:28.375 INFO:teuthology.orchestra.run.smithi115.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-09-20T22:28:28.375 INFO:teuthology.orchestra.run.smithi115.stderr:Mon IP `172.21.15.115` is in CIDR network `172.21.0.0/20` 2024-09-20T22:28:28.375 INFO:teuthology.orchestra.run.smithi115.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-20T22:28:28.376 INFO:teuthology.orchestra.run.smithi115.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-09-20T22:28:28.474 INFO:teuthology.orchestra.run.smithi115.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-09-20T22:28:28.816 INFO:teuthology.orchestra.run.smithi115.stderr:/bin/podman: Getting image source signatures 2024-09-20T22:28:28.816 INFO:teuthology.orchestra.run.smithi115.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-09-20T22:28:28.816 INFO:teuthology.orchestra.run.smithi115.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-09-20T22:28:28.816 INFO:teuthology.orchestra.run.smithi115.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-09-20T22:28:28.816 INFO:teuthology.orchestra.run.smithi115.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-09-20T22:28:28.817 INFO:teuthology.orchestra.run.smithi115.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-09-20T22:28:28.817 INFO:teuthology.orchestra.run.smithi115.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-09-20T22:28:28.984 INFO:teuthology.orchestra.run.smithi115.stderr:/bin/podman: Writing manifest to image destination 2024-09-20T22:28:29.137 INFO:teuthology.orchestra.run.smithi115.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-09-20T22:28:29.545 INFO:teuthology.orchestra.run.smithi115.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-09-20T22:28:30.119 INFO:teuthology.orchestra.run.smithi115.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-09-20T22:28:30.119 INFO:teuthology.orchestra.run.smithi115.stderr:Extracting ceph user uid/gid from container image... 2024-09-20T22:28:30.527 INFO:teuthology.orchestra.run.smithi115.stderr:stat: 167 167 2024-09-20T22:28:32.085 INFO:teuthology.orchestra.run.smithi115.stderr:Creating initial keys... 2024-09-20T22:28:35.757 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-authtool: AQAS9+1mTkpkLhAA0SMmsoCsZjkzscYzY0t15g== 2024-09-20T22:28:42.811 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-authtool: AQAa9+1mxU7dKxAAEnNt3XYmfa3JEFkPJCX9OA== 2024-09-20T22:28:43.708 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph-authtool: AQAb9+1mYW4tJRAAm+zZA2oXbvCjZm1hRJwm2w== 2024-09-20T22:28:44.357 INFO:teuthology.orchestra.run.smithi115.stderr:Creating initial monmap... 2024-09-20T22:28:44.805 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T22:28:44.806 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-09-20T22:28:44.806 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:28:44.806 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T22:28:45.403 INFO:teuthology.orchestra.run.smithi115.stderr:monmaptool for a [v2:172.21.15.115:3300,v1:172.21.15.115:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T22:28:45.403 INFO:teuthology.orchestra.run.smithi115.stderr:setting min_mon_release = octopus 2024-09-20T22:28:45.403 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/monmaptool: set fsid to 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:28:45.404 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T22:28:45.404 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-20T22:28:45.404 INFO:teuthology.orchestra.run.smithi115.stderr:Creating mon... 2024-09-20T22:28:46.310 INFO:teuthology.orchestra.run.smithi115.stderr:create mon.a on 2024-09-20T22:28:46.528 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-20T22:28:46.679 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9fe0f662-779f-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-9fe0f662-779f-11ef-baf5-efdc52797490.target. 2024-09-20T22:28:46.679 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-9fe0f662-779f-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-9fe0f662-779f-11ef-baf5-efdc52797490.target. 2024-09-20T22:28:46.971 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Failed to reset failed state of unit ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.a.service: Unit ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.a.service not loaded. 2024-09-20T22:28:46.979 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Created symlink /etc/systemd/system/ceph-9fe0f662-779f-11ef-baf5-efdc52797490.target.wants/ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.a.service → /etc/systemd/system/ceph-9fe0f662-779f-11ef-baf5-efdc52797490@.service. 2024-09-20T22:28:48.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:48 smithi115 ceph-mon[23382]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T22:28:48.810 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: disabled 2024-09-20T22:28:48.818 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2024-09-20T22:28:48.818 INFO:teuthology.orchestra.run.smithi115.stderr:firewalld.service is not enabled 2024-09-20T22:28:48.818 INFO:teuthology.orchestra.run.smithi115.stderr:Not possible to enable service . firewalld.service is not available 2024-09-20T22:28:48.819 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for mon to start... 2024-09-20T22:28:48.819 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for mon... 2024-09-20T22:28:49.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:48 smithi115 bash[23364]: 304a1846851a3f52a487960481e1e942445423a2053df754fa4f95494ea8136c 2024-09-20T22:28:49.047 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:48 smithi115 systemd[1]: Started Ceph mon.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:28:49.307 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: cluster: 2024-09-20T22:28:49.307 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: id: 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:28:49.307 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-09-20T22:28:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2024-09-20T22:28:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: services: 2024-09-20T22:28:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.845652s) 2024-09-20T22:28:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mgr: no daemons active 2024-09-20T22:28:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-09-20T22:28:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2024-09-20T22:28:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: data: 2024-09-20T22:28:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-09-20T22:28:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-09-20T22:28:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-09-20T22:28:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: pgs: 2024-09-20T22:28:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2024-09-20T22:28:49.900 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:49 smithi115 ceph-mon[23382]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T22:28:49.900 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:49 smithi115 ceph-mon[23382]: monmap e1: 1 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} 2024-09-20T22:28:49.900 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:49 smithi115 ceph-mon[23382]: fsmap 2024-09-20T22:28:49.900 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:49 smithi115 ceph-mon[23382]: osdmap e1: 0 total, 0 up, 0 in 2024-09-20T22:28:49.900 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:49 smithi115 ceph-mon[23382]: mgrmap e1: no daemons active 2024-09-20T22:28:49.900 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:49 smithi115 ceph-mon[23382]: from='client.? 172.21.15.115:0/3301611439' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-20T22:28:49.902 INFO:teuthology.orchestra.run.smithi115.stderr:mon is available 2024-09-20T22:28:49.902 INFO:teuthology.orchestra.run.smithi115.stderr:Assimilating anything we can from ceph.conf... 2024-09-20T22:28:50.400 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2024-09-20T22:28:50.400 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: [global] 2024-09-20T22:28:50.400 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: fsid = 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:28:50.400 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.115:3300,v1:172.21.15.115:6789] 2024-09-20T22:28:50.400 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-09-20T22:28:50.400 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-09-20T22:28:50.400 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-09-20T22:28:50.401 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-09-20T22:28:50.401 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2024-09-20T22:28:50.401 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: [mgr] 2024-09-20T22:28:50.401 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-09-20T22:28:50.401 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-09-20T22:28:50.401 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2024-09-20T22:28:50.401 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: [osd] 2024-09-20T22:28:50.401 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-09-20T22:28:50.401 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-09-20T22:28:50.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:50 smithi115 ceph-mon[23382]: from='client.? 172.21.15.115:0/136360123' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T22:28:50.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:50 smithi115 ceph-mon[23382]: from='client.? 172.21.15.115:0/136360123' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-20T22:28:50.979 INFO:teuthology.orchestra.run.smithi115.stderr:Generating new minimal ceph.conf... 2024-09-20T22:28:51.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:51 smithi115 ceph-mon[23382]: from='client.? 172.21.15.115:0/3673968625' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:28:52.032 INFO:teuthology.orchestra.run.smithi115.stderr:Restarting the monitor... 2024-09-20T22:28:52.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:52 smithi115 systemd[1]: Stopping Ceph mon.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:28:52.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:52 smithi115 bash[23600]: Error: no container with name or ID "ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon.a" found: no such container 2024-09-20T22:28:52.488 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:52 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a[23378]: 2024-09-20T22:28:52.232+0000 7fd26a9d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:28:52.488 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:52 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a[23378]: 2024-09-20T22:28:52.232+0000 7fd26a9d3700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-20T22:28:52.800 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:52 smithi115 podman[23610]: 2024-09-20 22:28:52.487241938 +0000 UTC m=+0.405753207 container died 304a1846851a3f52a487960481e1e942445423a2053df754fa4f95494ea8136c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , name=centos-stream, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, release=754, version=8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc.) 2024-09-20T22:28:53.108 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:52 smithi115 podman[23610]: 2024-09-20 22:28:52.898233456 +0000 UTC m=+0.816744725 container cleanup 304a1846851a3f52a487960481e1e942445423a2053df754fa4f95494ea8136c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, distribution-scope=public, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.buildah.version=1.19.8, architecture=x86_64, release=754, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vcs-type=git, name=centos-stream, io.openshift.expose-services=) 2024-09-20T22:28:53.108 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:52 smithi115 bash[23610]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a 2024-09-20T22:28:53.108 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:52 smithi115 bash[23637]: Error: no container with name or ID "ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon.a" found: no such container 2024-09-20T22:28:53.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 podman[23622]: 2024-09-20 22:28:53.107354618 +0000 UTC m=+0.615396676 container remove 304a1846851a3f52a487960481e1e942445423a2053df754fa4f95494ea8136c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, io.k8s.display-name=CentOS Stream 8, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.component=centos-stream-container, RELEASE=HEAD, release=754, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.openshift.expose-services=, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0) 2024-09-20T22:28:53.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-20T22:28:53.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 systemd[1]: Stopped Ceph mon.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:28:53.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 systemd[1]: Starting Ceph mon.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:28:53.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 podman[23701]: 2024-09-20 22:28:53.232558118 +0000 UTC m=+0.016001890 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-09-20T22:28:53.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 podman[23701]: 2024-09-20 22:28:53.341543083 +0000 UTC m=+0.124986845 container create 0b3311e89569035e1e8559114a97e1f09352b0d04b427b43bb1ba6e8065f199a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, maintainer=Guillaume Abrioux , name=centos-stream, ceph=True, vcs-type=git, distribution-scope=public, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, version=8, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-09-20T22:28:53.623 INFO:teuthology.orchestra.run.smithi115.stderr:Setting mon public_network to 172.21.0.0/20 2024-09-20T22:28:53.746 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 podman[23701]: 2024-09-20 22:28:53.552475998 +0000 UTC m=+0.335919766 container init 0b3311e89569035e1e8559114a97e1f09352b0d04b427b43bb1ba6e8065f199a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, version=8, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, ceph=True, RELEASE=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc.) 2024-09-20T22:28:53.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 podman[23701]: 2024-09-20 22:28:53.555456836 +0000 UTC m=+0.338900595 container start 0b3311e89569035e1e8559114a97e1f09352b0d04b427b43bb1ba6e8065f199a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, version=8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git) 2024-09-20T22:28:53.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T22:28:53.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-09-20T22:28:53.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: pidfile_write: ignore empty --pid-file 2024-09-20T22:28:53.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: load: jerasure load: lrc 2024-09-20T22:28:53.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: RocksDB version: 6.15.5 2024-09-20T22:28:53.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-20T22:28:53.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Compile date Apr 18 2022 2024-09-20T22:28:53.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: DB SUMMARY 2024-09-20T22:28:53.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: DB Session ID: QO0RYBGP0HIJ6CNFMYTG 2024-09-20T22:28:53.747 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: CURRENT file: CURRENT 2024-09-20T22:28:53.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T22:28:53.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-20T22:28:53.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-20T22:28:53.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82657 ; 2024-09-20T22:28:53.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.error_if_exists: 0 2024-09-20T22:28:53.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.create_if_missing: 0 2024-09-20T22:28:53.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.paranoid_checks: 1 2024-09-20T22:28:53.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T22:28:53.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.env: 0x555a650c6860 2024-09-20T22:28:53.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.fs: Posix File System 2024-09-20T22:28:53.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.info_log: 0x555a65a7ddc0 2024-09-20T22:28:53.748 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.statistics: (nil) 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.use_fsync: 0 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_log_file_size: 0 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.allow_fallocate: 1 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.use_direct_reads: 0 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.db_log_dir: 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-20T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T22:28:53.750 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T22:28:53.750 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T22:28:53.750 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T22:28:53.750 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T22:28:53.750 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T22:28:53.750 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T22:28:53.751 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T22:28:53.751 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.write_buffer_manager: 0x555a65b6e240 2024-09-20T22:28:53.751 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T22:28:53.751 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-20T22:28:53.751 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T22:28:53.751 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.unordered_write: 0 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.row_cache: None 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.wal_filter: None 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T22:28:53.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.preserve_deletes: 0 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.two_write_queues: 0 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.atomic_flush: 0 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.log_readahead_size: 0 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T22:28:53.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T22:28:53.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_background_jobs: 2 2024-09-20T22:28:53.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_background_compactions: -1 2024-09-20T22:28:53.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_subcompactions: 1 2024-09-20T22:28:53.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T22:28:53.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T22:28:53.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T22:28:53.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T22:28:53.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T22:28:53.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T22:28:53.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T22:28:53.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T22:28:53.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_open_files: -1 2024-09-20T22:28:53.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_background_flushes: -1 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Compression algorithms supported: 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: kZSTD supported: 0 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: kXpressCompression supported: 0 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: kLZ4Compression supported: 1 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: kBZip2Compression supported: 0 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: kZlibCompression supported: 1 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: kSnappyCompression supported: 1 2024-09-20T22:28:53.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.merge_operator: 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compaction_filter: None 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555a65a49d00) 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout: cache_index_and_filter_blocks: 1 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout: pin_top_level_index_and_filter: 1 2024-09-20T22:28:53.756 INFO:journalctl@ceph.mon.a.smithi115.stdout: index_type: 0 2024-09-20T22:28:53.757 INFO:journalctl@ceph.mon.a.smithi115.stdout: data_block_index_type: 0 2024-09-20T22:28:53.757 INFO:journalctl@ceph.mon.a.smithi115.stdout: index_shortening: 1 2024-09-20T22:28:53.757 INFO:journalctl@ceph.mon.a.smithi115.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T22:28:53.757 INFO:journalctl@ceph.mon.a.smithi115.stdout: hash_index_allow_collision: 1 2024-09-20T22:28:53.757 INFO:journalctl@ceph.mon.a.smithi115.stdout: checksum: 1 2024-09-20T22:28:53.757 INFO:journalctl@ceph.mon.a.smithi115.stdout: no_block_cache: 0 2024-09-20T22:28:53.757 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache: 0x555a65ab4170 2024-09-20T22:28:53.757 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache_name: BinnedLRUCache 2024-09-20T22:28:53.757 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache_options: 2024-09-20T22:28:53.757 INFO:journalctl@ceph.mon.a.smithi115.stdout: capacity : 536870912 2024-09-20T22:28:53.757 INFO:journalctl@ceph.mon.a.smithi115.stdout: num_shard_bits : 4 2024-09-20T22:28:53.757 INFO:journalctl@ceph.mon.a.smithi115.stdout: strict_capacity_limit : 0 2024-09-20T22:28:53.757 INFO:journalctl@ceph.mon.a.smithi115.stdout: high_pri_pool_ratio: 0.000 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache_compressed: (nil) 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: persistent_cache: (nil) 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_size: 4096 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_size_deviation: 10 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_restart_interval: 16 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: index_block_restart_interval: 1 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: metadata_block_size: 4096 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: partition_filters: 0 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: use_delta_encoding: 1 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: whole_key_filtering: 1 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: verify_compression: 0 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: read_amp_bytes_per_bit: 0 2024-09-20T22:28:53.758 INFO:journalctl@ceph.mon.a.smithi115.stdout: format_version: 4 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout: enable_index_compression: 1 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_align: 0 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compression: NoCompression 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.num_levels: 7 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T22:28:53.759 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T22:28:53.760 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T22:28:53.761 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.arena_block_size: 4194304 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T22:28:53.762 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T22:28:53.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T22:28:53.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.table_properties_collectors: 2024-09-20T22:28:53.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.inplace_update_support: 0 2024-09-20T22:28:53.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T22:28:53.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T22:28:53.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T22:28:53.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T22:28:53.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.bloom_locality: 0 2024-09-20T22:28:53.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.max_successive_merges: 0 2024-09-20T22:28:53.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T22:28:53.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T22:28:53.763 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T22:28:53.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T22:28:53.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.ttl: 2592000 2024-09-20T22:28:53.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T22:28:53.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.enable_blob_files: false 2024-09-20T22:28:53.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.min_blob_size: 0 2024-09-20T22:28:53.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T22:28:53.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T22:28:53.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T22:28:53.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T22:28:53.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T22:28:53.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-09-20T22:28:53.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-09-20T22:28:53.764 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726871333581687, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726871333582376, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79207, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77470, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71779, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726871333, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "39618671-acfa-43c3-bd7e-cf98cbe798a0", "db_session_id": "QO0RYBGP0HIJ6CNFMYTG"}} 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726871333582686, "job": 1, "event": "recovery_finished"} 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x555a65a9aa80 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: DB pointer 0x555a65aaa000 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** DB Stats ** 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T22:28:53.765 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T22:28:53.766 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T22:28:53.766 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-20T22:28:53.766 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T22:28:53.766 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2024-09-20T22:28:53.766 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** Compaction Stats [default] ** 2024-09-20T22:28:53.766 INFO:journalctl@ceph.mon.a.smithi115.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-20T22:28:53.766 INFO:journalctl@ceph.mon.a.smithi115.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T22:28:53.766 INFO:journalctl@ceph.mon.a.smithi115.stdout: L0 2/0 79.11 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-20T22:28:53.766 INFO:journalctl@ceph.mon.a.smithi115.stdout: Sum 2/0 79.11 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-20T22:28:53.766 INFO:journalctl@ceph.mon.a.smithi115.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-20T22:28:53.766 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2024-09-20T22:28:53.766 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** Compaction Stats [default] ** 2024-09-20T22:28:53.766 INFO:journalctl@ceph.mon.a.smithi115.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative compaction: 0.00 GB write, 25.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval compaction: 0.00 GB write, 25.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T22:28:53.767 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** Compaction Stats [default] ** 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: L0 2/0 79.11 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: Sum 2/0 79.11 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** Compaction Stats [default] ** 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 163.9 0.00 0.00 1 0.000 0 0 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T22:28:53.768 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative compaction: 0.00 GB write, 25.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: starting mon.a rank 0 at public addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] at bind addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: mon.a@-1(???) e1 preinit fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: mon.a@-1(???).mds e1 new map 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: mon.a@-1(???).mds e1 print_map 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout: e1 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T22:28:53.769 INFO:journalctl@ceph.mon.a.smithi115.stdout: legacy client fscid: -1 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout: No filesystems configured 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: expand_channel_meta expand map: {default=false} 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: expand_channel_meta from 'false' to 'false' 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: expand_channel_meta expanded map: {default=false} 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: expand_channel_meta expand map: {default=info} 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: expand_channel_meta from 'info' to 'info' 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: expand_channel_meta expanded map: {default=info} 2024-09-20T22:28:53.770 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: expand_channel_meta expand map: {default=daemon} 2024-09-20T22:28:53.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: expand_channel_meta from 'daemon' to 'daemon' 2024-09-20T22:28:53.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: expand_channel_meta expanded map: {default=daemon} 2024-09-20T22:28:53.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: expand_channel_meta expand map: {default=debug} 2024-09-20T22:28:53.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: expand_channel_meta from 'debug' to 'debug' 2024-09-20T22:28:53.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: expand_channel_meta expanded map: {default=debug} 2024-09-20T22:28:53.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 bash[23701]: 0b3311e89569035e1e8559114a97e1f09352b0d04b427b43bb1ba6e8065f199a 2024-09-20T22:28:53.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 systemd[1]: Started Ceph mon.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:28:53.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T22:28:53.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: monmap e1: 1 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0]} 2024-09-20T22:28:53.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: fsmap 2024-09-20T22:28:53.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: osdmap e1: 0 total, 0 up, 0 in 2024-09-20T22:28:53.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:53 smithi115 ceph-mon[23720]: mgrmap e1: no daemons active 2024-09-20T22:28:54.755 INFO:teuthology.orchestra.run.smithi115.stderr:Wrote config to /etc/ceph/ceph.conf 2024-09-20T22:28:54.755 INFO:teuthology.orchestra.run.smithi115.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-20T22:28:54.755 INFO:teuthology.orchestra.run.smithi115.stderr:Creating mgr... 2024-09-20T22:28:54.755 INFO:teuthology.orchestra.run.smithi115.stderr:Verifying port 9283 ... 2024-09-20T22:28:54.928 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Failed to reset failed state of unit ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.y.service: Unit ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.y.service not loaded. 2024-09-20T22:28:54.935 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: Created symlink /etc/systemd/system/ceph-9fe0f662-779f-11ef-baf5-efdc52797490.target.wants/ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-9fe0f662-779f-11ef-baf5-efdc52797490@.service. 2024-09-20T22:28:55.341 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:55 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1935951502' entity='client.admin' 2024-09-20T22:28:55.341 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:55 smithi115 podman[23933]: 2024-09-20 22:28:55.301808577 +0000 UTC m=+0.118539623 container create 8bc4016f60564c377027c6313f699beb3eaa9ede83bc60263385958b64fea915 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, maintainer=Guillaume Abrioux , name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, ceph=True, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, build-date=2022-05-03T08:36:31.336870) 2024-09-20T22:28:55.597 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:55 smithi115 podman[23933]: 2024-09-20 22:28:55.504525519 +0000 UTC m=+0.321256564 container init 8bc4016f60564c377027c6313f699beb3eaa9ede83bc60263385958b64fea915 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, vendor=Red Hat, Inc., release=754, architecture=x86_64, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, distribution-scope=public, com.redhat.component=centos-stream-container, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.buildah.version=1.19.8) 2024-09-20T22:28:55.597 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:55 smithi115 podman[23933]: 2024-09-20 22:28:55.507437442 +0000 UTC m=+0.324168478 container start 8bc4016f60564c377027c6313f699beb3eaa9ede83bc60263385958b64fea915 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vcs-type=git, ceph=True, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, distribution-scope=public, vendor=Red Hat, Inc., io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-09-20T22:28:55.611 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: disabled 2024-09-20T22:28:55.619 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2024-09-20T22:28:55.619 INFO:teuthology.orchestra.run.smithi115.stderr:firewalld.service is not enabled 2024-09-20T22:28:55.620 INFO:teuthology.orchestra.run.smithi115.stderr:Not possible to enable service . firewalld.service is not available 2024-09-20T22:28:55.628 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: disabled 2024-09-20T22:28:55.635 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2024-09-20T22:28:55.635 INFO:teuthology.orchestra.run.smithi115.stderr:firewalld.service is not enabled 2024-09-20T22:28:55.635 INFO:teuthology.orchestra.run.smithi115.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-09-20T22:28:55.636 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for mgr to start... 2024-09-20T22:28:55.636 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for mgr... 2024-09-20T22:28:55.870 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:55 smithi115 bash[23933]: 8bc4016f60564c377027c6313f699beb3eaa9ede83bc60263385958b64fea915 2024-09-20T22:28:55.870 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:55 smithi115 systemd[1]: Started Ceph mgr.y for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:28:55.871 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:55.653+0000 7f1f109ba000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:28:55.871 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:55.733+0000 7f1f109ba000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:28:56.138 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:55.912+0000 7f1f109ba000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:28:56.138 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:56.000+0000 7f1f109ba000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:28:56.138 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:56.136+0000 7f1f109ba000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:28:56.183 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2024-09-20T22:28:56.183 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2024-09-20T22:28:56.184 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsid": "9fe0f662-779f-11ef-baf5-efdc52797490", 2024-09-20T22:28:56.184 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "health": { 2024-09-20T22:28:56.184 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-20T22:28:56.184 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "checks": {}, 2024-09-20T22:28:56.184 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mutes": [] 2024-09-20T22:28:56.184 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:56.184 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-20T22:28:56.184 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum": [ 2024-09-20T22:28:56.184 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 0 2024-09-20T22:28:56.184 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2024-09-20T22:28:56.184 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-20T22:28:56.184 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "a" 2024-09-20T22:28:56.184 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2024-09-20T22:28:56.185 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-09-20T22:28:56.185 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "monmap": { 2024-09-20T22:28:56.185 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:28:56.185 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-09-20T22:28:56.185 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-20T22:28:56.185 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:56.185 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osdmap": { 2024-09-20T22:28:56.186 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:28:56.186 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-20T22:28:56.186 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-20T22:28:56.186 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-20T22:28:56.186 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-20T22:28:56.186 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-20T22:28:56.186 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-20T22:28:56.186 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:56.186 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgmap": { 2024-09-20T22:28:56.186 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-20T22:28:56.186 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-20T22:28:56.186 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-20T22:28:56.187 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-20T22:28:56.187 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-20T22:28:56.187 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-20T22:28:56.187 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-20T22:28:56.187 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-20T22:28:56.187 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:56.187 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsmap": { 2024-09-20T22:28:56.187 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:28:56.187 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-20T22:28:56.188 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-20T22:28:56.188 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:56.188 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-20T22:28:56.188 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "available": false, 2024-09-20T22:28:56.188 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-20T22:28:56.188 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modules": [ 2024-09-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "iostat", 2024-09-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "nfs", 2024-09-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "restful" 2024-09-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2024-09-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2024-09-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "servicemap": { 2024-09-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modified": "2024-09-20T22:28:48.459856+0000", 2024-09-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2024-09-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-20T22:28:56.189 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2024-09-20T22:28:56.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:56 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/805057695' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T22:28:56.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:56.351+0000 7f1f109ba000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:28:56.723 INFO:teuthology.orchestra.run.smithi115.stderr:mgr not available, waiting (1/15)... 2024-09-20T22:28:56.776 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:56.686+0000 7f1f109ba000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:28:57.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:56.774+0000 7f1f109ba000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:28:57.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:56.941+0000 7f1f109ba000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:28:58.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:57 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:57.994+0000 7f1f109ba000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:28:58.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:58 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:58.085+0000 7f1f109ba000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:28:59.106 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:58 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:58.996+0000 7f1f109ba000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:28:59.204 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2024-09-20T22:28:59.204 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2024-09-20T22:28:59.204 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsid": "9fe0f662-779f-11ef-baf5-efdc52797490", 2024-09-20T22:28:59.204 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "health": { 2024-09-20T22:28:59.204 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-20T22:28:59.204 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "checks": {}, 2024-09-20T22:28:59.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mutes": [] 2024-09-20T22:28:59.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:59.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-20T22:28:59.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum": [ 2024-09-20T22:28:59.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 0 2024-09-20T22:28:59.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2024-09-20T22:28:59.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-20T22:28:59.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "a" 2024-09-20T22:28:59.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2024-09-20T22:28:59.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-09-20T22:28:59.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "monmap": { 2024-09-20T22:28:59.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:28:59.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-09-20T22:28:59.206 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-20T22:28:59.206 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:59.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osdmap": { 2024-09-20T22:28:59.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:28:59.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-20T22:28:59.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-20T22:28:59.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-20T22:28:59.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-20T22:28:59.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-20T22:28:59.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-20T22:28:59.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:59.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgmap": { 2024-09-20T22:28:59.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-20T22:28:59.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsmap": { 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-20T22:28:59.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "available": false, 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modules": [ 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "iostat", 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "nfs", 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "restful" 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "servicemap": { 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modified": "2024-09-20T22:28:48.459856+0000", 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-20T22:28:59.209 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2024-09-20T22:28:59.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:59.313+0000 7f1f109ba000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:28:59.725 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:28:59 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2375660593' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T22:28:59.726 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:59.448+0000 7f1f109ba000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:28:59.726 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:59.537+0000 7f1f109ba000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:28:59.726 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:59.634+0000 7f1f109ba000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:28:59.726 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:59.723+0000 7f1f109ba000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:28:59.737 INFO:teuthology.orchestra.run.smithi115.stderr:mgr not available, waiting (2/15)... 2024-09-20T22:29:00.065 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:59.874+0000 7f1f109ba000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:29:00.065 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:28:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:28:59.967+0000 7f1f109ba000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:29:00.345 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:00 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:00.063+0000 7f1f109ba000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:29:00.346 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:00 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:00.343+0000 7f1f109ba000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:29:00.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:00 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:00.431+0000 7f1f109ba000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:29:00.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:00 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:00.520+0000 7f1f109ba000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:29:01.853 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:01.577+0000 7f1f109ba000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:29:01.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-mon[23720]: Activating manager daemon y 2024-09-20T22:29:01.853 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-mon[23720]: mgrmap e2: y(active, starting, since 0.00292198s) 2024-09-20T22:29:01.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-mon[23720]: from='mgr.14100 172.21.15.115:0/1985517653' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:29:01.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-mon[23720]: from='mgr.14100 172.21.15.115:0/1985517653' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:29:01.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-mon[23720]: from='mgr.14100 172.21.15.115:0/1985517653' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:29:01.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-mon[23720]: from='mgr.14100 172.21.15.115:0/1985517653' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:29:01.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-mon[23720]: from='mgr.14100 172.21.15.115:0/1985517653' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:29:01.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-mon[23720]: Manager daemon y is now available 2024-09-20T22:29:01.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-mon[23720]: from='mgr.14100 172.21.15.115:0/1985517653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:29:01.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-mon[23720]: from='mgr.14100 172.21.15.115:0/1985517653' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:29:01.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-mon[23720]: from='mgr.14100 172.21.15.115:0/1985517653' entity='mgr.y' 2024-09-20T22:29:01.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-mon[23720]: from='mgr.14100 172.21.15.115:0/1985517653' entity='mgr.y' 2024-09-20T22:29:01.854 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:01 smithi115 ceph-mon[23720]: from='mgr.14100 172.21.15.115:0/1985517653' entity='mgr.y' 2024-09-20T22:29:02.200 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2024-09-20T22:29:02.200 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2024-09-20T22:29:02.200 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsid": "9fe0f662-779f-11ef-baf5-efdc52797490", 2024-09-20T22:29:02.200 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "health": { 2024-09-20T22:29:02.200 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-20T22:29:02.200 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "checks": {}, 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mutes": [] 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum": [ 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 0 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "a" 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "monmap": { 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-20T22:29:02.201 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:02.202 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osdmap": { 2024-09-20T22:29:02.202 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:29:02.202 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-20T22:29:02.202 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-20T22:29:02.202 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-20T22:29:02.202 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-20T22:29:02.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-20T22:29:02.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-20T22:29:02.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:02.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgmap": { 2024-09-20T22:29:02.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-20T22:29:02.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-20T22:29:02.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-20T22:29:02.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-20T22:29:02.205 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-20T22:29:02.206 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-20T22:29:02.206 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-20T22:29:02.206 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-20T22:29:02.206 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:02.206 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsmap": { 2024-09-20T22:29:02.206 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:29:02.206 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-20T22:29:02.206 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-20T22:29:02.206 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:02.206 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-20T22:29:02.206 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "available": false, 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modules": [ 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "iostat", 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "nfs", 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "restful" 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "servicemap": { 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modified": "2024-09-20T22:28:48.459856+0000", 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:02.207 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-20T22:29:02.208 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2024-09-20T22:29:02.775 INFO:teuthology.orchestra.run.smithi115.stderr:mgr not available, waiting (3/15)... 2024-09-20T22:29:02.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:02 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1272157992' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T22:29:02.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:02 smithi115 ceph-mon[23720]: mgrmap e3: y(active, since 1.00598s) 2024-09-20T22:29:04.903 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:04 smithi115 ceph-mon[23720]: mgrmap e4: y(active, since 2s) 2024-09-20T22:29:05.438 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2024-09-20T22:29:05.438 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2024-09-20T22:29:05.438 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsid": "9fe0f662-779f-11ef-baf5-efdc52797490", 2024-09-20T22:29:05.438 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "health": { 2024-09-20T22:29:05.438 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-20T22:29:05.438 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "checks": {}, 2024-09-20T22:29:05.438 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mutes": [] 2024-09-20T22:29:05.438 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:05.438 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-20T22:29:05.438 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum": [ 2024-09-20T22:29:05.439 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 0 2024-09-20T22:29:05.439 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2024-09-20T22:29:05.439 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-20T22:29:05.439 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "a" 2024-09-20T22:29:05.439 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2024-09-20T22:29:05.439 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-09-20T22:29:05.439 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "monmap": { 2024-09-20T22:29:05.439 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:29:05.439 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-09-20T22:29:05.439 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-20T22:29:05.439 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:05.439 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osdmap": { 2024-09-20T22:29:05.439 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:29:05.440 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-20T22:29:05.440 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-20T22:29:05.440 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-20T22:29:05.440 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-20T22:29:05.440 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-20T22:29:05.440 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-20T22:29:05.440 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:05.443 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgmap": { 2024-09-20T22:29:05.443 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-20T22:29:05.443 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-20T22:29:05.443 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-20T22:29:05.443 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-20T22:29:05.443 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-20T22:29:05.443 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-20T22:29:05.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-20T22:29:05.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-20T22:29:05.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:05.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "fsmap": { 2024-09-20T22:29:05.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:29:05.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-20T22:29:05.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-20T22:29:05.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:05.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-20T22:29:05.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "available": true, 2024-09-20T22:29:05.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-20T22:29:05.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modules": [ 2024-09-20T22:29:05.445 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "iostat", 2024-09-20T22:29:05.445 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "nfs", 2024-09-20T22:29:05.445 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "restful" 2024-09-20T22:29:05.445 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ], 2024-09-20T22:29:05.445 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2024-09-20T22:29:05.445 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:05.445 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "servicemap": { 2024-09-20T22:29:05.445 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T22:29:05.445 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "modified": "2024-09-20T22:28:48.459856+0000", 2024-09-20T22:29:05.445 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "services": {} 2024-09-20T22:29:05.445 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: }, 2024-09-20T22:29:05.445 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-20T22:29:05.445 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2024-09-20T22:29:05.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:05 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/943735603' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T22:29:06.031 INFO:teuthology.orchestra.run.smithi115.stderr:mgr is available 2024-09-20T22:29:06.641 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2024-09-20T22:29:06.642 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: [global] 2024-09-20T22:29:06.642 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: fsid = 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:29:06.642 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-09-20T22:29:06.642 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-09-20T22:29:06.642 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-09-20T22:29:06.642 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-09-20T22:29:06.642 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2024-09-20T22:29:06.642 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: [mgr] 2024-09-20T22:29:06.642 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-09-20T22:29:06.642 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 2024-09-20T22:29:06.642 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: [osd] 2024-09-20T22:29:06.642 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-09-20T22:29:06.643 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-09-20T22:29:06.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:06 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3936552908' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T22:29:06.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:06 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3936552908' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-20T22:29:07.188 INFO:teuthology.orchestra.run.smithi115.stderr:Enabling cephadm module... 2024-09-20T22:29:08.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:07 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/860709569' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-20T22:29:08.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:07 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ignoring --setuser ceph since I am not root 2024-09-20T22:29:08.155 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:07 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ignoring --setgroup ceph since I am not root 2024-09-20T22:29:08.155 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:08 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:08.041+0000 7f7235356000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:29:08.155 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:08 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:08.121+0000 7f7235356000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:29:08.500 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:08 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:08.295+0000 7f7235356000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:29:08.500 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:08 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:08.383+0000 7f7235356000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:29:08.754 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:08 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:08.517+0000 7f7235356000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:29:08.754 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:08 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:08.735+0000 7f7235356000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:29:09.073 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:08 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/860709569' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-20T22:29:09.073 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:08 smithi115 ceph-mon[23720]: mgrmap e5: y(active, since 6s) 2024-09-20T22:29:09.172 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2024-09-20T22:29:09.172 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 5, 2024-09-20T22:29:09.172 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "available": true, 2024-09-20T22:29:09.172 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "active_name": "y", 2024-09-20T22:29:09.173 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_standby": 0 2024-09-20T22:29:09.173 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2024-09-20T22:29:09.337 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:09 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:09.071+0000 7f7235356000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:29:09.337 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:09 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:09.160+0000 7f7235356000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:29:09.337 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:09 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:09.335+0000 7f7235356000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:29:09.758 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for the mgr to restart... 2024-09-20T22:29:09.758 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for mgr epoch 5... 2024-09-20T22:29:10.008 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:09 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/4177947369' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T22:29:10.904 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:10 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:10.402+0000 7f7235356000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:29:10.904 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:10 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:10.492+0000 7f7235356000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:29:11.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:11 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:11.383+0000 7f7235356000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:29:12.006 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:11 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:11.687+0000 7f7235356000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:29:12.007 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:11 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:11.818+0000 7f7235356000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:29:12.007 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:11 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:11.906+0000 7f7235356000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:29:12.007 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:12.004+0000 7f7235356000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:29:12.335 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:12.092+0000 7f7235356000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:29:12.335 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:12.241+0000 7f7235356000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:29:12.335 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:12.332+0000 7f7235356000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:29:12.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:12.429+0000 7f7235356000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:29:13.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:12.708+0000 7f7235356000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:29:13.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:12.794+0000 7f7235356000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:29:13.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:12.885+0000 7f7235356000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:29:14.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:13.946+0000 7f7235356000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:29:14.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:14 smithi115 ceph-mon[23720]: Active manager daemon y restarted 2024-09-20T22:29:14.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:14 smithi115 ceph-mon[23720]: Activating manager daemon y 2024-09-20T22:29:14.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:14 smithi115 ceph-mon[23720]: osdmap e2: 0 total, 0 up, 0 in 2024-09-20T22:29:15.607 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2024-09-20T22:29:15.607 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-09-20T22:29:15.607 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "initialized": true 2024-09-20T22:29:15.607 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2024-09-20T22:29:15.680 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:29:15] ENGINE Bus STARTING 2024-09-20T22:29:15.680 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: mgrmap e6: y(active, starting, since 0.64567s) 2024-09-20T22:29:15.680 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:29:15.680 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:29:15.680 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:29:15.680 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:29:15.680 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:29:15.681 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: Manager daemon y is now available 2024-09-20T22:29:15.681 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' 2024-09-20T22:29:15.681 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' 2024-09-20T22:29:15.681 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:29:15.681 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:29:15.681 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:29:15.681 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:29:15.681 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:29:15.974 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:29:15] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:29:15.974 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:29:15] ENGINE Bus STARTED 2024-09-20T22:29:16.164 INFO:teuthology.orchestra.run.smithi115.stderr:mgr epoch 5 is available 2024-09-20T22:29:16.164 INFO:teuthology.orchestra.run.smithi115.stderr:Setting orchestrator backend to cephadm... 2024-09-20T22:29:16.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:16 smithi115 ceph-mon[23720]: [20/Sep/2024:22:29:15] ENGINE Bus STARTING 2024-09-20T22:29:16.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:16 smithi115 ceph-mon[23720]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T22:29:16.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:16 smithi115 ceph-mon[23720]: mgrmap e7: y(active, since 1.65027s) 2024-09-20T22:29:16.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:16 smithi115 ceph-mon[23720]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T22:29:16.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:16 smithi115 ceph-mon[23720]: [20/Sep/2024:22:29:15] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:29:16.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:16 smithi115 ceph-mon[23720]: [20/Sep/2024:22:29:15] ENGINE Bus STARTED 2024-09-20T22:29:16.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:16 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' 2024-09-20T22:29:16.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:16 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:29:17.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:17 smithi115 ceph-mon[23720]: mgrmap e8: y(active, since 2s) 2024-09-20T22:29:17.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:17 smithi115 ceph-mon[23720]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:17.771 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:17 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' 2024-09-20T22:29:17.772 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:17 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:29:18.060 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: value unchanged 2024-09-20T22:29:18.643 INFO:teuthology.orchestra.run.smithi115.stderr:Generating ssh key... 2024-09-20T22:29:19.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-mon[23720]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:19.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' 2024-09-20T22:29:19.797 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' 2024-09-20T22:29:19.797 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: Generating public/private rsa key pair. 2024-09-20T22:29:19.797 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: Your identification has been saved in /tmp/tmp83buwue6/key. 2024-09-20T22:29:19.797 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: Your public key has been saved in /tmp/tmp83buwue6/key.pub. 2024-09-20T22:29:19.797 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: The key fingerprint is: 2024-09-20T22:29:19.797 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: SHA256:TA6tvnwY9c3mA57jH+RyLA5RuxbPPDrL+6xvGaHQXrA ceph-9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:29:19.797 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: The key's randomart image is: 2024-09-20T22:29:19.797 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: +---[RSA 3072]----+ 2024-09-20T22:29:19.797 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: | | 2024-09-20T22:29:19.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: | . . | 2024-09-20T22:29:19.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: | . o..o | 2024-09-20T22:29:19.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: | *o.E.o | 2024-09-20T22:29:19.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: | ..S+o=.. | 2024-09-20T22:29:19.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: | .. .=X= | 2024-09-20T22:29:19.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: | .o..==Xo | 2024-09-20T22:29:19.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: | ....==*=o | 2024-09-20T22:29:19.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: | o. .O@=. | 2024-09-20T22:29:19.798 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: +----[SHA256]-----+ 2024-09-20T22:29:20.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:20 smithi115 ceph-mon[23720]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:20.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:20 smithi115 ceph-mon[23720]: Generating ssh key... 2024-09-20T22:29:20.666 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq44BWsyQv+0mz6MxqjoxtNX9jmKfwjtdaDy23EisFCa4RhElzYzGCBmgyx2M8ZUNYaKMdVAXZ2HlOJzEhyUikIgtWMzZZjRezDVoWqYBO2CQFjq8tLhqpAWsK/2LyIWnQmALrWBdEyTtpAGPWb/OemCe+Yn/8fnPdoWwYBE45C1oAfc17thCLorSU0cssFYASlu1RXTNYk+nhbLTAh2NYRKRFz1GragsCk/ZOm8GDPFi1FVFbmpjHN3iRfmauZoZl5vg3dkna1Mew5Bj4BIOknDopId8SH+GGNu4iZ3gLvIl3+xXQSpMhBDOdUn8tyG4V9pYu5XWxnlXHadB/bk3pBDKF11RKdAzqo44pYqsk0RUsfGQXtih+QmHv9Ceag+BLzZGhdcsyycVZdF8OMNxiuGrg4FBQ7+kG4o8ZN8I/0raJK0jI7YYZrsgrO1gulw50dyhhY/sWl3JZAJ8TozZQ4JWF6be1E5+GqlAFf4+Vvxf7FYDjDagyqnJn7AC8vVE= ceph-9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:29:21.256 INFO:teuthology.orchestra.run.smithi115.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-20T22:29:21.257 INFO:teuthology.orchestra.run.smithi115.stderr:Adding key to root@localhost authorized_keys... 2024-09-20T22:29:21.264 INFO:teuthology.orchestra.run.smithi115.stderr:Adding host smithi115... 2024-09-20T22:29:21.894 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:21 smithi115 ceph-mon[23720]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:22.772 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: Added host 'smithi115' with addr '172.21.15.115' 2024-09-20T22:29:22.814 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:22 smithi115 ceph-mon[23720]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi115", "addr": "172.21.15.115", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:23.355 INFO:teuthology.orchestra.run.smithi115.stderr:Deploying unmanaged mon service... 2024-09-20T22:29:23.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:23 smithi115 ceph-mon[23720]: Deploying cephadm binary to smithi115 2024-09-20T22:29:23.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:23 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' 2024-09-20T22:29:23.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:23 smithi115 ceph-mon[23720]: Added host smithi115 2024-09-20T22:29:23.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:23 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:29:24.002 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: Scheduled mon update... 2024-09-20T22:29:24.888 INFO:teuthology.orchestra.run.smithi115.stderr:Deploying unmanaged mgr service... 2024-09-20T22:29:25.135 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:24 smithi115 ceph-mon[23720]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:25.135 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:24 smithi115 ceph-mon[23720]: Saving service mon spec with placement count:5 2024-09-20T22:29:25.135 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:24 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' 2024-09-20T22:29:25.561 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-09-20T22:29:26.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:26 smithi115 ceph-mon[23720]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:26.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:26 smithi115 ceph-mon[23720]: Saving service mgr spec with placement count:2 2024-09-20T22:29:26.605 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:26 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' 2024-09-20T22:29:28.267 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:28 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2267700894' entity='client.admin' 2024-09-20T22:29:29.123 INFO:teuthology.orchestra.run.smithi115.stderr:Enabling the dashboard module... 2024-09-20T22:29:29.343 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:29 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1505555978' entity='client.admin' 2024-09-20T22:29:29.343 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:29 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' 2024-09-20T22:29:29.343 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:29 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' 2024-09-20T22:29:30.496 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:30 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2582699814' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-20T22:29:30.496 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:30 smithi115 ceph-mon[23720]: from='mgr.14120 172.21.15.115:0/3984581546' entity='mgr.y' 2024-09-20T22:29:30.496 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:30 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ignoring --setuser ceph since I am not root 2024-09-20T22:29:30.496 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:30 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ignoring --setgroup ceph since I am not root 2024-09-20T22:29:30.496 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:30 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:30.416+0000 7f222e8a9000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:29:30.496 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:30 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:30.493+0000 7f222e8a9000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:29:30.763 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:30 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:30.672+0000 7f222e8a9000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:29:30.763 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:30 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:30.761+0000 7f222e8a9000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:29:31.022 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:30 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:30.900+0000 7f222e8a9000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:29:31.324 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:31 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2582699814' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-20T22:29:31.325 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:31 smithi115 ceph-mon[23720]: mgrmap e9: y(active, since 16s) 2024-09-20T22:29:31.325 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:31 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:31.120+0000 7f222e8a9000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:29:31.588 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:31 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:31.474+0000 7f222e8a9000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:29:31.588 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:31 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:31.562+0000 7f222e8a9000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:29:31.590 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2024-09-20T22:29:31.590 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "epoch": 9, 2024-09-20T22:29:31.590 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "available": true, 2024-09-20T22:29:31.590 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "active_name": "y", 2024-09-20T22:29:31.590 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "num_standby": 0 2024-09-20T22:29:31.590 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2024-09-20T22:29:31.878 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:31 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:31.729+0000 7f222e8a9000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:29:32.180 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for the mgr to restart... 2024-09-20T22:29:32.180 INFO:teuthology.orchestra.run.smithi115.stderr:Waiting for mgr epoch 9... 2024-09-20T22:29:32.505 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:32 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/72452739' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T22:29:33.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:32 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:32.788+0000 7f222e8a9000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:29:33.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:32 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:32.878+0000 7f222e8a9000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:29:34.055 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:33 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:33.761+0000 7f222e8a9000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:29:34.399 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:34.064+0000 7f222e8a9000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:29:34.399 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:34.202+0000 7f222e8a9000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:29:34.399 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:34.295+0000 7f222e8a9000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:29:34.399 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:34.396+0000 7f222e8a9000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:29:34.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:34.490+0000 7f222e8a9000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:29:34.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:34.643+0000 7f222e8a9000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:29:35.119 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:34.739+0000 7f222e8a9000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:29:35.120 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:34.835+0000 7f222e8a9000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:29:35.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:35.117+0000 7f222e8a9000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:29:35.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:35.209+0000 7f222e8a9000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:29:35.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:35.299+0000 7f222e8a9000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:29:36.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:36 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:29:36.364+0000 7f222e8a9000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:29:36.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:36 smithi115 ceph-mon[23720]: Active manager daemon y restarted 2024-09-20T22:29:36.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:36 smithi115 ceph-mon[23720]: Activating manager daemon y 2024-09-20T22:29:36.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:36 smithi115 ceph-mon[23720]: osdmap e3: 0 total, 0 up, 0 in 2024-09-20T22:29:37.443 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: { 2024-09-20T22:29:37.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-09-20T22:29:37.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: "initialized": true 2024-09-20T22:29:37.444 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: } 2024-09-20T22:29:37.803 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:29:37] ENGINE Bus STARTING 2024-09-20T22:29:37.803 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:29:37] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:29:37.804 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:29:37] ENGINE Bus STARTED 2024-09-20T22:29:37.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-mon[23720]: mgrmap e10: y(active, starting, since 0.0525467s) 2024-09-20T22:29:37.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:29:37.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:29:37.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:29:37.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:29:37.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:29:37.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-mon[23720]: Manager daemon y is now available 2024-09-20T22:29:37.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:29:37.804 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:29:37.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:29:37.805 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:29:38.081 INFO:teuthology.orchestra.run.smithi115.stderr:mgr epoch 9 is available 2024-09-20T22:29:38.081 INFO:teuthology.orchestra.run.smithi115.stderr:Generating a dashboard self-signed certificate... 2024-09-20T22:29:38.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:38 smithi115 ceph-mon[23720]: mgrmap e11: y(active, since 1.06643s) 2024-09-20T22:29:38.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:38 smithi115 ceph-mon[23720]: [20/Sep/2024:22:29:37] ENGINE Bus STARTING 2024-09-20T22:29:38.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:38 smithi115 ceph-mon[23720]: [20/Sep/2024:22:29:37] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:29:38.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:38 smithi115 ceph-mon[23720]: [20/Sep/2024:22:29:37] ENGINE Bus STARTED 2024-09-20T22:29:38.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:38 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:38.749 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: Self-signed certificate created 2024-09-20T22:29:39.404 INFO:teuthology.orchestra.run.smithi115.stderr:Creating initial admin user... 2024-09-20T22:29:39.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:39 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:39.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:39 smithi115 ceph-mon[23720]: mgrmap e12: y(active, since 2s) 2024-09-20T22:29:39.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:39 smithi115 ceph-mon[23720]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:39.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:39 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:39.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:39 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:39.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:39 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:40.290 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$q/5D.qrPAAcVjd/gacWNb.YvVJWUdP/NkYUeD3Mc.gI5I0mjla6CS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726871380, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-20T22:29:41.633 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:41 smithi115 ceph-mon[23720]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:41.633 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:41 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:41.635 INFO:teuthology.orchestra.run.smithi115.stderr:Fetching dashboard port number... 2024-09-20T22:29:42.279 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: 8443 2024-09-20T22:29:42.631 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:42 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1436658344' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-20T22:29:42.943 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: disabled 2024-09-20T22:29:42.951 INFO:teuthology.orchestra.run.smithi115.stderr:systemctl: inactive 2024-09-20T22:29:42.951 INFO:teuthology.orchestra.run.smithi115.stderr:firewalld.service is not enabled 2024-09-20T22:29:42.951 INFO:teuthology.orchestra.run.smithi115.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-20T22:29:42.954 INFO:teuthology.orchestra.run.smithi115.stderr:Ceph Dashboard is now available at: 2024-09-20T22:29:42.954 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-20T22:29:42.954 INFO:teuthology.orchestra.run.smithi115.stderr: URL: https://smithi115.front.sepia.ceph.com:8443/ 2024-09-20T22:29:42.954 INFO:teuthology.orchestra.run.smithi115.stderr: User: admin 2024-09-20T22:29:42.954 INFO:teuthology.orchestra.run.smithi115.stderr: Password: s0nbw2h01o 2024-09-20T22:29:42.954 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-20T22:29:42.955 INFO:teuthology.orchestra.run.smithi115.stderr:Enabling autotune for osd_memory_target 2024-09-20T22:29:44.531 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:44 smithi115 ceph-mon[23720]: mgrmap e13: y(active, since 6s) 2024-09-20T22:29:44.905 INFO:teuthology.orchestra.run.smithi115.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-09-20T22:29:45.479 INFO:teuthology.orchestra.run.smithi115.stderr:You can access the Ceph CLI with: 2024-09-20T22:29:45.479 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-20T22:29:45.479 INFO:teuthology.orchestra.run.smithi115.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-20T22:29:45.479 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-20T22:29:45.479 INFO:teuthology.orchestra.run.smithi115.stderr:Please consider enabling telemetry to help improve Ceph: 2024-09-20T22:29:45.479 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-20T22:29:45.479 INFO:teuthology.orchestra.run.smithi115.stderr: ceph telemetry on 2024-09-20T22:29:45.479 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-20T22:29:45.479 INFO:teuthology.orchestra.run.smithi115.stderr:For more information see: 2024-09-20T22:29:45.480 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-20T22:29:45.480 INFO:teuthology.orchestra.run.smithi115.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-20T22:29:45.480 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-20T22:29:45.480 INFO:teuthology.orchestra.run.smithi115.stderr:Bootstrap complete. 2024-09-20T22:29:45.511 INFO:tasks.cephadm:Fetching config... 2024-09-20T22:29:45.511 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-20T22:29:45.511 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-20T22:29:45.527 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-20T22:29:45.527 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-20T22:29:45.527 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-20T22:29:45.583 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-20T22:29:45.583 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-20T22:29:45.583 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-20T22:29:45.649 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-20T22:29:45.649 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-20T22:29:45.649 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-20T22:29:45.704 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-20T22:29:45.704 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq44BWsyQv+0mz6MxqjoxtNX9jmKfwjtdaDy23EisFCa4RhElzYzGCBmgyx2M8ZUNYaKMdVAXZ2HlOJzEhyUikIgtWMzZZjRezDVoWqYBO2CQFjq8tLhqpAWsK/2LyIWnQmALrWBdEyTtpAGPWb/OemCe+Yn/8fnPdoWwYBE45C1oAfc17thCLorSU0cssFYASlu1RXTNYk+nhbLTAh2NYRKRFz1GragsCk/ZOm8GDPFi1FVFbmpjHN3iRfmauZoZl5vg3dkna1Mew5Bj4BIOknDopId8SH+GGNu4iZ3gLvIl3+xXQSpMhBDOdUn8tyG4V9pYu5XWxnlXHadB/bk3pBDKF11RKdAzqo44pYqsk0RUsfGQXtih+QmHv9Ceag+BLzZGhdcsyycVZdF8OMNxiuGrg4FBQ7+kG4o8ZN8I/0raJK0jI7YYZrsgrO1gulw50dyhhY/sWl3JZAJ8TozZQ4JWF6be1E5+GqlAFf4+Vvxf7FYDjDagyqnJn7AC8vVE= ceph-9fe0f662-779f-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T22:29:45.777 INFO:teuthology.orchestra.run.smithi115.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq44BWsyQv+0mz6MxqjoxtNX9jmKfwjtdaDy23EisFCa4RhElzYzGCBmgyx2M8ZUNYaKMdVAXZ2HlOJzEhyUikIgtWMzZZjRezDVoWqYBO2CQFjq8tLhqpAWsK/2LyIWnQmALrWBdEyTtpAGPWb/OemCe+Yn/8fnPdoWwYBE45C1oAfc17thCLorSU0cssFYASlu1RXTNYk+nhbLTAh2NYRKRFz1GragsCk/ZOm8GDPFi1FVFbmpjHN3iRfmauZoZl5vg3dkna1Mew5Bj4BIOknDopId8SH+GGNu4iZ3gLvIl3+xXQSpMhBDOdUn8tyG4V9pYu5XWxnlXHadB/bk3pBDKF11RKdAzqo44pYqsk0RUsfGQXtih+QmHv9Ceag+BLzZGhdcsyycVZdF8OMNxiuGrg4FBQ7+kG4o8ZN8I/0raJK0jI7YYZrsgrO1gulw50dyhhY/sWl3JZAJ8TozZQ4JWF6be1E5+GqlAFf4+Vvxf7FYDjDagyqnJn7AC8vVE= ceph-9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:29:45.789 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq44BWsyQv+0mz6MxqjoxtNX9jmKfwjtdaDy23EisFCa4RhElzYzGCBmgyx2M8ZUNYaKMdVAXZ2HlOJzEhyUikIgtWMzZZjRezDVoWqYBO2CQFjq8tLhqpAWsK/2LyIWnQmALrWBdEyTtpAGPWb/OemCe+Yn/8fnPdoWwYBE45C1oAfc17thCLorSU0cssFYASlu1RXTNYk+nhbLTAh2NYRKRFz1GragsCk/ZOm8GDPFi1FVFbmpjHN3iRfmauZoZl5vg3dkna1Mew5Bj4BIOknDopId8SH+GGNu4iZ3gLvIl3+xXQSpMhBDOdUn8tyG4V9pYu5XWxnlXHadB/bk3pBDKF11RKdAzqo44pYqsk0RUsfGQXtih+QmHv9Ceag+BLzZGhdcsyycVZdF8OMNxiuGrg4FBQ7+kG4o8ZN8I/0raJK0jI7YYZrsgrO1gulw50dyhhY/sWl3JZAJ8TozZQ4JWF6be1E5+GqlAFf4+Vvxf7FYDjDagyqnJn7AC8vVE= ceph-9fe0f662-779f-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T22:29:45.843 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq44BWsyQv+0mz6MxqjoxtNX9jmKfwjtdaDy23EisFCa4RhElzYzGCBmgyx2M8ZUNYaKMdVAXZ2HlOJzEhyUikIgtWMzZZjRezDVoWqYBO2CQFjq8tLhqpAWsK/2LyIWnQmALrWBdEyTtpAGPWb/OemCe+Yn/8fnPdoWwYBE45C1oAfc17thCLorSU0cssFYASlu1RXTNYk+nhbLTAh2NYRKRFz1GragsCk/ZOm8GDPFi1FVFbmpjHN3iRfmauZoZl5vg3dkna1Mew5Bj4BIOknDopId8SH+GGNu4iZ3gLvIl3+xXQSpMhBDOdUn8tyG4V9pYu5XWxnlXHadB/bk3pBDKF11RKdAzqo44pYqsk0RUsfGQXtih+QmHv9Ceag+BLzZGhdcsyycVZdF8OMNxiuGrg4FBQ7+kG4o8ZN8I/0raJK0jI7YYZrsgrO1gulw50dyhhY/sWl3JZAJ8TozZQ4JWF6be1E5+GqlAFf4+Vvxf7FYDjDagyqnJn7AC8vVE= ceph-9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:29:45.856 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-20T22:29:46.038 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:45 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2509405221' entity='client.admin' 2024-09-20T22:29:48.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:48.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:29:48.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:48.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:29:48.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:48.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:48 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2743544910' entity='client.admin' 2024-09-20T22:29:48.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:29:48.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:48.977 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-20T22:29:48.977 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-20T22:29:51.519 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi153 2024-09-20T22:29:51.519 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-20T22:29:51.519 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.conf 2024-09-20T22:29:51.538 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-20T22:29:51.539 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:29:51.597 INFO:tasks.cephadm:Adding host smithi153 to orchestrator... 2024-09-20T22:29:51.597 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch host add smithi153 2024-09-20T22:29:52.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:51 smithi115 ceph-mon[23720]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:52.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:52.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:29:52.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:29:52.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:29:52.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:51 smithi115 ceph-mon[23720]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:29:52.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:51 smithi115 ceph-mon[23720]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:29:52.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:52.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:54.348 INFO:teuthology.orchestra.run.smithi115.stdout:Added host 'smithi153' with addr '172.21.15.153' 2024-09-20T22:29:54.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:54 smithi115 ceph-mon[23720]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:29:54.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:54 smithi115 ceph-mon[23720]: Deploying cephadm binary to smithi153 2024-09-20T22:29:54.598 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:54 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:54.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:54 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:29:54.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:54 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:29:54.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:54 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:29:54.968 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-20T22:29:55.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:55 smithi115 ceph-mon[23720]: Added host smithi153 2024-09-20T22:29:55.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:55 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:55.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:55 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:56.840 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:29:56.840 INFO:teuthology.orchestra.run.smithi115.stdout:[{"addr": "172.21.15.115", "hostname": "smithi115", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}] 2024-09-20T22:29:57.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:57 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:29:57.437 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-20T22:29:57.438 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd crush tunables default 2024-09-20T22:29:58.262 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:58 smithi115 ceph-mon[23720]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:29:58.262 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:29:58 smithi115 ceph-mon[23720]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:30:00.156 INFO:teuthology.orchestra.run.smithi115.stderr:adjusted tunables profile to default 2024-09-20T22:30:00.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:00 smithi115 ceph-mon[23720]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:00.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:00 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:00.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:00 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:00.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:00 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:30:00.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:00 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:00.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:00 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/4232080238' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T22:30:00.796 INFO:tasks.cephadm:Adding mon.a on smithi115 2024-09-20T22:30:00.796 INFO:tasks.cephadm:Adding mon.c on smithi115 2024-09-20T22:30:00.797 INFO:tasks.cephadm:Adding mon.b on smithi153 2024-09-20T22:30:00.797 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch apply mon '3;smithi115:172.21.15.115=a;smithi115:[v2:172.21.15.115:3301,v1:172.21.15.115:6790]=c;smithi153:172.21.15.153=b' 2024-09-20T22:30:01.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:01 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/4232080238' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T22:30:01.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:01 smithi115 ceph-mon[23720]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T22:30:02.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:02 smithi115 ceph-mon[23720]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:02.934 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mon update... 2024-09-20T22:30:03.583 DEBUG:teuthology.orchestra.run.smithi115:mon.c> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.c.service 2024-09-20T22:30:03.586 DEBUG:teuthology.orchestra.run.smithi153:mon.b> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.b.service 2024-09-20T22:30:03.588 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T22:30:03.588 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T22:30:04.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:03 smithi115 ceph-mon[23720]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:04.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:03 smithi115 ceph-mon[23720]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi115:172.21.15.115=a;smithi115:[v2:172.21.15.115:3301,v1:172.21.15.115:6790]=c;smithi153:172.21.15.153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:03 smithi115 ceph-mon[23720]: Saving service mon spec with placement smithi115:172.21.15.115=a;smithi115:[v2:172.21.15.115:3301,v1:172.21.15.115:6790]=c;smithi153:172.21.15.153=b;count:3 2024-09-20T22:30:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:03 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:03 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:03 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:03 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:03 smithi115 ceph-mon[23720]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:30:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:03 smithi115 ceph-mon[23720]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:30:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:03 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:03 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:03 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:30:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:03 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:05.027 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:04 smithi115 ceph-mon[23720]: Deploying daemon mon.c on smithi115 2024-09-20T22:30:05.415 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:30:05.415 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"9fe0f662-779f-11ef-baf5-efdc52797490","modified":"2024-09-20T22:28:44.690924Z","created":"2024-09-20T22:28:44.690924Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T22:30:05.417 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-09-20T22:30:06.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:05 smithi115 ceph-mon[23720]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:06.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:05 smithi115 ceph-mon[23720]: from='client.? 172.21.15.153:0/4004882766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:30:06.641 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:06 smithi115 systemd[1]: Started Ceph mon.c for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:30:07.003 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T22:30:07.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T22:30:11.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:11.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: mon.a calling monitor election 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: mon.c calling monitor election 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: monmap e2: 2 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: fsmap 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T22:30:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: mgrmap e13: y(active, since 35s) 2024-09-20T22:30:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: overall HEALTH_OK 2024-09-20T22:30:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:30:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:11.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:11.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:30:11.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: mon.a calling monitor election 2024-09-20T22:30:11.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:11.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:11.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:11.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: mon.c calling monitor election 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: monmap e2: 2 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: fsmap 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: mgrmap e13: y(active, since 35s) 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: overall HEALTH_OK 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:30:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:11 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:11.908 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:30:11.908 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":2,"fsid":"9fe0f662-779f-11ef-baf5-efdc52797490","modified":"2024-09-20T22:30:06.567437Z","created":"2024-09-20T22:28:44.690924Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3301","nonce":0},{"type":"v1","addr":"172.21.15.115:6790","nonce":0}]},"addr":"172.21.15.115:6790/0","public_addr":"172.21.15.115:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-20T22:30:11.908 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 2 2024-09-20T22:30:12.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:12 smithi115 ceph-mon[23720]: Deploying daemon mon.b on smithi153 2024-09-20T22:30:12.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:12 smithi115 ceph-mon[23720]: from='client.? 172.21.15.153:0/422399621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:30:12.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:12 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:12.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:12 smithi115 ceph-mon[28216]: Deploying daemon mon.b on smithi153 2024-09-20T22:30:12.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:12 smithi115 ceph-mon[28216]: from='client.? 172.21.15.153:0/422399621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:30:12.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:12 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:13.644 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T22:30:13.644 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T22:30:13.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:13 smithi115 ceph-mon[23720]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:13.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:13 smithi115 ceph-mon[28216]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:15.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:15 smithi115 ceph-mon[23720]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:15.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:15 smithi115 ceph-mon[28216]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:16.699 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:30:16.699 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":2,"fsid":"9fe0f662-779f-11ef-baf5-efdc52797490","modified":"2024-09-20T22:30:06.567437Z","created":"2024-09-20T22:28:44.690924Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3301","nonce":0},{"type":"v1","addr":"172.21.15.115:6790","nonce":0}]},"addr":"172.21.15.115:6790/0","public_addr":"172.21.15.115:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-20T22:30:16.700 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 2 2024-09-20T22:30:16.808 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:16 smithi153 ceph-mon[24571]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-09-20T22:30:17.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:16 smithi153 bash[24518]: 62b3afaddd2673d388680c2ee1c24cf55f746373873fbd005cf106f0ba40f2e3 2024-09-20T22:30:17.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:16 smithi153 systemd[1]: Started Ceph mon.b for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:30:18.375 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T22:30:18.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T22:30:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:30:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: mon.a calling monitor election 2024-09-20T22:30:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: mon.c calling monitor election 2024-09-20T22:30:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: mon.b calling monitor election 2024-09-20T22:30:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-20T22:30:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2024-09-20T22:30:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: fsmap 2024-09-20T22:30:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T22:30:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: mgrmap e13: y(active, since 45s) 2024-09-20T22:30:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: overall HEALTH_OK 2024-09-20T22:30:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:30:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: mon.a calling monitor election 2024-09-20T22:30:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: mon.c calling monitor election 2024-09-20T22:30:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: mon.b calling monitor election 2024-09-20T22:30:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-20T22:30:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2024-09-20T22:30:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: fsmap 2024-09-20T22:30:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T22:30:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: mgrmap e13: y(active, since 45s) 2024-09-20T22:30:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: overall HEALTH_OK 2024-09-20T22:30:22.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:22.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:22.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:22.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:30:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: mon.a calling monitor election 2024-09-20T22:30:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: mon.c calling monitor election 2024-09-20T22:30:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:30:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: mon.b calling monitor election 2024-09-20T22:30:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-20T22:30:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: monmap e3: 3 mons at {a=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0]} 2024-09-20T22:30:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: fsmap 2024-09-20T22:30:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T22:30:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: mgrmap e13: y(active, since 45s) 2024-09-20T22:30:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: overall HEALTH_OK 2024-09-20T22:30:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:23.152 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:30:23.152 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":3,"fsid":"9fe0f662-779f-11ef-baf5-efdc52797490","modified":"2024-09-20T22:30:16.783199Z","created":"2024-09-20T22:28:44.690924Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3301","nonce":0},{"type":"v1","addr":"172.21.15.115:6790","nonce":0}]},"addr":"172.21.15.115:6790/0","public_addr":"172.21.15.115:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-20T22:30:23.154 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 3 2024-09-20T22:30:23.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:23.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:30:23.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:23.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:30:23.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T22:30:23.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:30:23.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:30:23.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:23.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: Reconfiguring daemon mon.a on smithi115 2024-09-20T22:30:23.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:23 smithi153 ceph-mon[24571]: from='client.? 172.21.15.153:0/2814352531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:30:23.840 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-20T22:30:23.841 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph config generate-minimal-conf 2024-09-20T22:30:23.865 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:23.865 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.865 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:30:23.865 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:23.866 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.866 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:30:23.866 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.866 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.866 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.866 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T22:30:23.866 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:30:23.866 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:30:23.866 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:23.866 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: Reconfiguring daemon mon.a on smithi115 2024-09-20T22:30:23.866 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[23720]: from='client.? 172.21.15.153:0/2814352531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:30:23.866 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: Reconfiguring daemon mon.a on smithi115 2024-09-20T22:30:23.867 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:23 smithi115 ceph-mon[28216]: from='client.? 172.21.15.153:0/2814352531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:30:25.561 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:25 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:25.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:25 smithi115 ceph-mon[23720]: Reconfiguring mon.c (monmap changed)... 2024-09-20T22:30:25.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:25 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:30:25.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:25 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:30:25.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:25 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:25.562 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:25 smithi115 ceph-mon[23720]: Reconfiguring daemon mon.c on smithi115 2024-09-20T22:30:25.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:25 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:25.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:25 smithi115 ceph-mon[28216]: Reconfiguring mon.c (monmap changed)... 2024-09-20T22:30:25.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:25 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:30:25.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:25 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:30:25.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:25 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:25.562 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:25 smithi115 ceph-mon[28216]: Reconfiguring daemon mon.c on smithi115 2024-09-20T22:30:25.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:25 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:25.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:25 smithi153 ceph-mon[24571]: Reconfiguring mon.c (monmap changed)... 2024-09-20T22:30:25.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:25 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:30:25.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:25 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:30:25.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:25 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:25.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:25 smithi153 ceph-mon[24571]: Reconfiguring daemon mon.c on smithi115 2024-09-20T22:30:26.092 INFO:teuthology.orchestra.run.smithi115.stdout:# minimal ceph.conf for 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:30:26.092 INFO:teuthology.orchestra.run.smithi115.stdout:[global] 2024-09-20T22:30:26.092 INFO:teuthology.orchestra.run.smithi115.stdout: fsid = 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:30:26.092 INFO:teuthology.orchestra.run.smithi115.stdout: mon_host = [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] 2024-09-20T22:30:26.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:26 smithi115 ceph-mon[23720]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:26.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:26 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3086652668' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:26.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:26 smithi115 ceph-mon[28216]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:26.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:26 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3086652668' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:26.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:26 smithi153 ceph-mon[24571]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:26.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:26 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3086652668' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:27.145 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-20T22:30:27.145 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-20T22:30:27.145 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T22:30:27.212 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-20T22:30:27.212 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:30:27.277 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-20T22:30:27.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T22:30:27.307 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-20T22:30:27.307 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:30:27.375 INFO:tasks.cephadm:Adding mgr.y on smithi115 2024-09-20T22:30:27.376 INFO:tasks.cephadm:Adding mgr.x on smithi153 2024-09-20T22:30:27.376 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch apply mgr '2;smithi115=y;smithi153=x' 2024-09-20T22:30:27.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:27 smithi153 ceph-mon[24571]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:27.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:27 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:27.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:27 smithi115 ceph-mon[23720]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:27.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:27 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:27.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:27 smithi115 ceph-mon[28216]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:27.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:27 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:28.868 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:28 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:28.868 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:28 smithi153 ceph-mon[24571]: Reconfiguring mon.b (monmap changed)... 2024-09-20T22:30:28.868 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:28 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:30:28.868 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:28 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:30:28.868 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:28 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:28.868 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:28 smithi153 ceph-mon[24571]: Reconfiguring daemon mon.b on smithi153 2024-09-20T22:30:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:28 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:28 smithi115 ceph-mon[23720]: Reconfiguring mon.b (monmap changed)... 2024-09-20T22:30:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:28 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:30:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:28 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:30:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:28 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:29.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:28 smithi115 ceph-mon[23720]: Reconfiguring daemon mon.b on smithi153 2024-09-20T22:30:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:28 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:28 smithi115 ceph-mon[28216]: Reconfiguring mon.b (monmap changed)... 2024-09-20T22:30:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:28 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:30:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:28 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:30:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:28 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:28 smithi115 ceph-mon[28216]: Reconfiguring daemon mon.b on smithi153 2024-09-20T22:30:29.168 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mgr update... 2024-09-20T22:30:29.736 DEBUG:teuthology.orchestra.run.smithi153:mgr.x> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.x.service 2024-09-20T22:30:29.738 INFO:tasks.cephadm:Deploying OSDs... 2024-09-20T22:30:29.738 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-20T22:30:29.739 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T22:30:29.758 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T22:30:29.758 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 2024-09-20T22:30:29.817 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T22:30:29.817 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:30:29.817 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-20T22:30:29.817 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:30:29.817 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:30:29.817 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-09-20 22:30:05.998681776 +0000 2024-09-20T22:30:29.817 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-09-20 22:27:53.293256954 +0000 2024-09-20T22:30:29.818 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-09-20 22:27:53.293256954 +0000 2024-09-20T22:30:29.818 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-09-20 22:27:53.293256954 +0000 2024-09-20T22:30:29.818 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T22:30:29.885 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-09-20T22:30:29.885 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-09-20T22:30:29.885 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000231215 s, 2.2 MB/s 2024-09-20T22:30:29.886 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T22:30:29.946 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 2024-09-20T22:30:30.005 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T22:30:30.006 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:30:30.006 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-20T22:30:30.006 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:30:30.006 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:30:30.006 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-09-20 22:30:05.999681756 +0000 2024-09-20T22:30:30.006 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-09-20 22:27:53.610250809 +0000 2024-09-20T22:30:30.006 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-09-20 22:27:53.610250809 +0000 2024-09-20T22:30:30.006 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-09-20 22:27:53.610250809 +0000 2024-09-20T22:30:30.006 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T22:30:30.069 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-09-20T22:30:30.069 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-09-20T22:30:30.069 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000208367 s, 2.5 MB/s 2024-09-20T22:30:30.071 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T22:30:30.128 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 2024-09-20T22:30:30.185 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T22:30:30.185 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:30:30.186 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-20T22:30:30.186 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:30:30.186 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:30:30.186 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-09-20 22:30:05.999681756 +0000 2024-09-20T22:30:30.186 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-09-20 22:27:53.947244277 +0000 2024-09-20T22:30:30.186 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-09-20 22:27:53.947244277 +0000 2024-09-20T22:30:30.186 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-09-20 22:27:53.947244277 +0000 2024-09-20T22:30:30.186 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T22:30:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-09-20T22:30:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-09-20T22:30:30.251 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000216026 s, 2.4 MB/s 2024-09-20T22:30:30.252 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T22:30:30.309 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 2024-09-20T22:30:30.365 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T22:30:30.365 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:30:30.365 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-20T22:30:30.365 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:30:30.366 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:30:30.366 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-09-20 22:30:05.999681756 +0000 2024-09-20T22:30:30.366 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-09-20 22:27:54.269238036 +0000 2024-09-20T22:30:30.366 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-09-20 22:27:54.269238036 +0000 2024-09-20T22:30:30.366 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: 2024-09-20 22:27:54.269238036 +0000 2024-09-20T22:30:30.366 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T22:30:30.427 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:30 smithi115 ceph-mon[28216]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:30.427 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:30 smithi115 ceph-mon[28216]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi115=y;smithi153=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:30.428 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:30 smithi115 ceph-mon[28216]: Saving service mgr spec with placement smithi115=y;smithi153=x;count:2 2024-09-20T22:30:30.428 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:30 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:30.428 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:30 smithi115 ceph-mon[23720]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:30.428 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:30 smithi115 ceph-mon[23720]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi115=y;smithi153=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:30.428 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:30 smithi115 ceph-mon[23720]: Saving service mgr spec with placement smithi115=y;smithi153=x;count:2 2024-09-20T22:30:30.428 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:30 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:30.430 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-09-20T22:30:30.430 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-09-20T22:30:30.430 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000223734 s, 2.3 MB/s 2024-09-20T22:30:30.431 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T22:30:30.489 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-20T22:30:30.489 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T22:30:30.505 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T22:30:30.506 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-09-20T22:30:30.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:30 smithi153 ceph-mon[24571]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:30.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:30 smithi153 ceph-mon[24571]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi115=y;smithi153=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:30.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:30 smithi153 ceph-mon[24571]: Saving service mgr spec with placement smithi115=y;smithi153=x;count:2 2024-09-20T22:30:30.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:30 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:30.543 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T22:30:30.543 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:30:30.543 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-20T22:30:30.543 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:30:30.544 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:30:30.544 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-20 22:30:15.466566568 +0000 2024-09-20T22:30:30.544 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-20 22:27:53.321613767 +0000 2024-09-20T22:30:30.544 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-20 22:27:53.321613767 +0000 2024-09-20T22:30:30.544 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-20 22:27:53.321613767 +0000 2024-09-20T22:30:30.544 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T22:30:30.607 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-20T22:30:30.607 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-20T22:30:30.607 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000187043 s, 2.7 MB/s 2024-09-20T22:30:30.608 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T22:30:30.664 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-09-20T22:30:30.719 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T22:30:30.719 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:30:30.719 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-20T22:30:30.719 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:30:30.719 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:30:30.720 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-20 22:30:15.467566550 +0000 2024-09-20T22:30:30.720 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-20 22:27:53.646607737 +0000 2024-09-20T22:30:30.720 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-20 22:27:53.646607737 +0000 2024-09-20T22:30:30.720 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-20 22:27:53.646607737 +0000 2024-09-20T22:30:30.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T22:30:30.783 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-20T22:30:30.783 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-20T22:30:30.783 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000203453 s, 2.5 MB/s 2024-09-20T22:30:30.784 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T22:30:30.840 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-09-20T22:30:30.896 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T22:30:30.896 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:30:30.897 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-20T22:30:30.897 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:30:30.897 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:30:30.897 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-20 22:30:15.467566550 +0000 2024-09-20T22:30:30.897 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-20 22:27:53.966601800 +0000 2024-09-20T22:30:30.897 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-20 22:27:53.966601800 +0000 2024-09-20T22:30:30.897 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-20 22:27:53.966601800 +0000 2024-09-20T22:30:30.897 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T22:30:30.961 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-20T22:30:30.962 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-20T22:30:30.962 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000218667 s, 2.3 MB/s 2024-09-20T22:30:30.963 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T22:30:31.019 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-09-20T22:30:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T22:30:31.074 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:30:31.074 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-20T22:30:31.075 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:30:31.075 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:30:31.075 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-20 22:30:15.468566531 +0000 2024-09-20T22:30:31.075 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-20 22:27:54.291595770 +0000 2024-09-20T22:30:31.075 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-20 22:27:54.291595770 +0000 2024-09-20T22:30:31.075 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-20 22:27:54.291595770 +0000 2024-09-20T22:30:31.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T22:30:31.138 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-20T22:30:31.138 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-20T22:30:31.139 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000215079 s, 2.4 MB/s 2024-09-20T22:30:31.139 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T22:30:31.195 INFO:tasks.cephadm:Deploying osd.0 on smithi115 with /dev/vg_nvme/lv_4... 2024-09-20T22:30:31.195 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T22:30:31.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[28216]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:31.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:31.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:31.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:31.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:31.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:31.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:30:31.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T22:30:31.751 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:30:31.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:31.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[28216]: Deploying daemon mgr.x on smithi153 2024-09-20T22:30:31.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[23720]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:31.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:31.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:31.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:31.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:31.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:31.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:30:31.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T22:30:31.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:30:31.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:31.753 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:31 smithi115 ceph-mon[23720]: Deploying daemon mgr.x on smithi153 2024-09-20T22:30:32.024 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:31 smithi153 ceph-mon[24571]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:32.025 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:31 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:32.025 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:31 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:32.025 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:31 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:32.025 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:31 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:32.025 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:31 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:32.025 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:31 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:30:32.025 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:31 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T22:30:32.025 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:31 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:30:32.025 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:31 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:32.025 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:31 smithi153 ceph-mon[24571]: Deploying daemon mgr.x on smithi153 2024-09-20T22:30:32.949 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:32 smithi153 systemd[1]: Started Ceph mgr.x for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:30:33.225 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:32.947+0000 7fae996f8000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:30:33.226 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:33.031+0000 7fae996f8000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:30:33.226 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:33.223+0000 7fae996f8000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:30:33.528 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:33.337+0000 7fae996f8000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:30:33.529 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:33.472+0000 7fae996f8000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:30:33.928 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:30:33.950 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi115:vg_nvme/lv_4 2024-09-20T22:30:33.959 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:33.691+0000 7fae996f8000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:30:34.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:33 smithi115 ceph-mon[28216]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:34.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:34.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:34.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:34.260 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:34.261 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:33 smithi115 ceph-mon[23720]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:34.261 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:34.261 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:34.261 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:34.261 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:34.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:33 smithi153 ceph-mon[24571]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:34.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:34.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:34.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:34.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:34.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:34 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:34.045+0000 7fae996f8000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:30:34.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:34 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:34.131+0000 7fae996f8000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:30:34.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:34 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:34.294+0000 7fae996f8000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:30:35.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:35.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:35.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[28216]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T22:30:35.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:30:35.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:30:35.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:35.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[28216]: Reconfiguring daemon mgr.y on smithi115 2024-09-20T22:30:35.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:35.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:35.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[23720]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T22:30:35.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:30:35.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:30:35.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:35.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:34 smithi115 ceph-mon[23720]: Reconfiguring daemon mgr.y on smithi115 2024-09-20T22:30:35.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:34 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:35.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:34 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:34 smithi153 ceph-mon[24571]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T22:30:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:34 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:30:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:34 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:30:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:34 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:34 smithi153 ceph-mon[24571]: Reconfiguring daemon mgr.y on smithi115 2024-09-20T22:30:35.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:35 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:35.339+0000 7fae996f8000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:30:35.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:35 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:35.428+0000 7fae996f8000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:30:36.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:35 smithi153 ceph-mon[24571]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:36.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:35 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:36.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:35 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:36.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:35 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:36.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:35 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:36.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:35 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:36.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:35 smithi115 ceph-mon[23720]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:36.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:35 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:36.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:35 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:36.358 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:35 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:36.359 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:35 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:36.359 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:35 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:35 smithi115 ceph-mon[28216]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:35 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:35 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:35 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:35 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:36.359 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:35 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:36.635 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:36 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:36.328+0000 7fae996f8000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:30:36.953 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:36 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:36.633+0000 7fae996f8000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:30:36.953 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:36 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:36.765+0000 7fae996f8000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:30:36.953 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:36 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:36.854+0000 7fae996f8000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:30:36.953 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:36 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:36.951+0000 7fae996f8000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:30:37.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:36 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:30:37.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:36 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:30:37.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:36 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:37.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:37.041+0000 7fae996f8000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:30:37.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:37.187+0000 7fae996f8000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:30:37.403 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:36 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:30:37.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:36 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:30:37.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:36 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:37.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:36 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:30:37.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:36 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:30:37.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:36 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:37.663 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:37.280+0000 7fae996f8000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:30:37.663 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:37.376+0000 7fae996f8000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:30:37.999 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:37.661+0000 7fae996f8000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:30:37.999 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:37.748+0000 7fae996f8000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:30:37.999 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:37.837+0000 7fae996f8000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:30:38.242 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:37 smithi115 ceph-mon[23720]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:38.242 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:37 smithi115 ceph-mon[23720]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:38.242 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:30:38.242 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:30:38.242 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:38.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:37 smithi115 ceph-mon[28216]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:38.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:37 smithi115 ceph-mon[28216]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:38.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:37 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:30:38.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:37 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:30:38.243 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:37 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:38.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:37 smithi153 ceph-mon[24571]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:38.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:37 smithi153 ceph-mon[24571]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:30:38.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:37 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:30:38.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:37 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:30:38.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:37 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:39.278 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:30:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:30:38.900+0000 7fae996f8000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:30:39.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[28216]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:39.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[28216]: Standby manager daemon x started 2024-09-20T22:30:39.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/3015061303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:30:39.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/3015061303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:30:39.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/3015061303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:30:39.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/3015061303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:30:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1483849103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c2e890c-711f-4957-a306-2f3c211ab807"}]: dispatch 2024-09-20T22:30:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1483849103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c2e890c-711f-4957-a306-2f3c211ab807"}]': finished 2024-09-20T22:30:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[28216]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T22:30:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:39.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[23720]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:39.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[23720]: Standby manager daemon x started 2024-09-20T22:30:39.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/3015061303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:30:39.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/3015061303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:30:39.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/3015061303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:30:39.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/3015061303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:30:39.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1483849103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c2e890c-711f-4957-a306-2f3c211ab807"}]: dispatch 2024-09-20T22:30:39.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1483849103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c2e890c-711f-4957-a306-2f3c211ab807"}]': finished 2024-09-20T22:30:39.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[23720]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T22:30:39.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:39 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:40.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:39 smithi153 ceph-mon[24571]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:40.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:39 smithi153 ceph-mon[24571]: Standby manager daemon x started 2024-09-20T22:30:40.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:39 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/3015061303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:30:40.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:39 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/3015061303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:30:40.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:39 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/3015061303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:30:40.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:39 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/3015061303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:30:40.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:39 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1483849103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c2e890c-711f-4957-a306-2f3c211ab807"}]: dispatch 2024-09-20T22:30:40.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:39 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1483849103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c2e890c-711f-4957-a306-2f3c211ab807"}]': finished 2024-09-20T22:30:40.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:39 smithi153 ceph-mon[24571]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T22:30:40.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:39 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:40.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:40 smithi115 ceph-mon[23720]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-20T22:30:40.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:40 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:30:40.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:40 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3522680753' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:30:40.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:40 smithi115 ceph-mon[28216]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-20T22:30:40.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:40 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:30:40.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:40 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3522680753' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:30:41.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:40 smithi153 ceph-mon[24571]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-20T22:30:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:40 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:30:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:40 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3522680753' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:30:41.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:41 smithi115 ceph-mon[23720]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:41.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:41 smithi115 ceph-mon[28216]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:42.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:41 smithi153 ceph-mon[24571]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:43.788 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:43 smithi115 ceph-mon[23720]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:43.788 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:43 smithi115 ceph-mon[28216]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:44.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:43 smithi153 ceph-mon[24571]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:45.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:45 smithi115 ceph-mon[23720]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:45.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T22:30:45.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:45.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:45 smithi115 ceph-mon[28216]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:45.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T22:30:45.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:46.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:45 smithi153 ceph-mon[24571]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:46.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T22:30:46.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:46.749 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:46 smithi115 ceph-mon[23720]: Deploying daemon osd.0 on smithi115 2024-09-20T22:30:46.750 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:46 smithi115 ceph-mon[28216]: Deploying daemon osd.0 on smithi115 2024-09-20T22:30:46.931 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:46 smithi153 ceph-mon[24571]: Deploying daemon osd.0 on smithi115 2024-09-20T22:30:47.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:47 smithi153 ceph-mon[24571]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:47.903 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:47 smithi115 ceph-mon[23720]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:47.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:47 smithi115 ceph-mon[28216]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:49.895 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:49 smithi115 ceph-mon[23720]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:49.895 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:49 smithi115 ceph-mon[28216]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:50.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:49 smithi153 ceph-mon[24571]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:51.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:51.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:51.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:51.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:51.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:51 smithi115 ceph-mon[28216]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:51.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:51.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:51.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:51.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:51.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:51 smithi115 ceph-mon[23720]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:51.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:51.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:51.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:51.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:51.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:51 smithi153 ceph-mon[24571]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:53.744 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 0 on host 'smithi115' 2024-09-20T22:30:54.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[28216]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:54.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:54.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:54.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[28216]: from='osd.0 [v2:172.21.15.115:6802/3107198721,v1:172.21.15.115:6803/3107198721]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T22:30:54.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[28216]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T22:30:54.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:54.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:54.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:54.016 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:54.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[23720]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:54.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:54.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:54.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[23720]: from='osd.0 [v2:172.21.15.115:6802/3107198721,v1:172.21.15.115:6803/3107198721]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T22:30:54.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[23720]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T22:30:54.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:54.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:54.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:54.016 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:53 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:54.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:53 smithi153 ceph-mon[24571]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:54.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:53 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:54.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:53 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:54.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:53 smithi153 ceph-mon[24571]: from='osd.0 [v2:172.21.15.115:6802/3107198721,v1:172.21.15.115:6803/3107198721]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T22:30:54.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:53 smithi153 ceph-mon[24571]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T22:30:54.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:53 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:54.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:53 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:30:54.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:53 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:30:54.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:53 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:30:54.390 DEBUG:teuthology.orchestra.run.smithi115:osd.0> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.0.service 2024-09-20T22:30:54.392 INFO:tasks.cephadm:Deploying osd.1 on smithi115 with /dev/vg_nvme/lv_3... 2024-09-20T22:30:54.392 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T22:30:55.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:55 smithi153 ceph-mon[24571]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T22:30:55.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:55 smithi153 ceph-mon[24571]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T22:30:55.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:55 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:55.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:55 smithi153 ceph-mon[24571]: from='osd.0 [v2:172.21.15.115:6802/3107198721,v1:172.21.15.115:6803/3107198721]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:30:55.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:55 smithi153 ceph-mon[24571]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:30:55.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:55 smithi115 ceph-mon[23720]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T22:30:55.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:55 smithi115 ceph-mon[23720]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T22:30:55.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:55 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:55.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:55 smithi115 ceph-mon[23720]: from='osd.0 [v2:172.21.15.115:6802/3107198721,v1:172.21.15.115:6803/3107198721]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:30:55.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:55 smithi115 ceph-mon[23720]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:30:55.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:55 smithi115 ceph-mon[28216]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T22:30:55.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:55 smithi115 ceph-mon[28216]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T22:30:55.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:55 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:55.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:55 smithi115 ceph-mon[28216]: from='osd.0 [v2:172.21.15.115:6802/3107198721,v1:172.21.15.115:6803/3107198721]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:30:55.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:55 smithi115 ceph-mon[28216]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:30:55.655 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:30:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0[31048]: 2024-09-20T22:30:55.196+0000 7fbfb829e700 -1 osd.0 0 waiting for initial osdmap 2024-09-20T22:30:55.655 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:30:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0[31048]: 2024-09-20T22:30:55.199+0000 7fbfb4c39700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:30:56.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:56 smithi115 ceph-mon[28216]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:56.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:56 smithi115 ceph-mon[28216]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-09-20T22:30:56.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:56 smithi115 ceph-mon[28216]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T22:30:56.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:56 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:56.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:56 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:56.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:56 smithi115 ceph-mon[23720]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:56.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:56 smithi115 ceph-mon[23720]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-09-20T22:30:56.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:56 smithi115 ceph-mon[23720]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T22:30:56.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:56 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:56.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:56 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:56.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:56 smithi153 ceph-mon[24571]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:56.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:56 smithi153 ceph-mon[24571]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-09-20T22:30:56.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:56 smithi153 ceph-mon[24571]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T22:30:56.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:56 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:56.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:56 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:57.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:57 smithi153 ceph-mon[24571]: purged_snaps scrub starts 2024-09-20T22:30:57.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:57 smithi153 ceph-mon[24571]: purged_snaps scrub ok 2024-09-20T22:30:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:57 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:57 smithi153 ceph-mon[24571]: osd.0 [v2:172.21.15.115:6802/3107198721,v1:172.21.15.115:6803/3107198721] boot 2024-09-20T22:30:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:57 smithi153 ceph-mon[24571]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T22:30:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:57 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:57.646 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:57 smithi115 ceph-mon[28216]: purged_snaps scrub starts 2024-09-20T22:30:57.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:57 smithi115 ceph-mon[28216]: purged_snaps scrub ok 2024-09-20T22:30:57.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:57 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:57.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:57 smithi115 ceph-mon[28216]: osd.0 [v2:172.21.15.115:6802/3107198721,v1:172.21.15.115:6803/3107198721] boot 2024-09-20T22:30:57.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:57 smithi115 ceph-mon[28216]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T22:30:57.647 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:57 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:57.647 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:57 smithi115 ceph-mon[23720]: purged_snaps scrub starts 2024-09-20T22:30:57.647 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:57 smithi115 ceph-mon[23720]: purged_snaps scrub ok 2024-09-20T22:30:57.647 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:57 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:57.647 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:57 smithi115 ceph-mon[23720]: osd.0 [v2:172.21.15.115:6802/3107198721,v1:172.21.15.115:6803/3107198721] boot 2024-09-20T22:30:57.647 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:57 smithi115 ceph-mon[23720]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T22:30:57.647 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:57 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:30:58.344 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:58 smithi115 ceph-mon[23720]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:58.344 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:58 smithi115 ceph-mon[23720]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T22:30:58.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:58 smithi115 ceph-mon[28216]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:58.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:58 smithi115 ceph-mon[28216]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T22:30:58.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:58 smithi153 ceph-mon[24571]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:30:58.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:58 smithi153 ceph-mon[24571]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T22:30:59.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[23720]: Detected new or changed devices on smithi115 2024-09-20T22:30:59.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:30:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[23720]: Adjusting osd_memory_target on smithi115 to 16064M 2024-09-20T22:30:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[23720]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:30:59.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[28216]: Detected new or changed devices on smithi115 2024-09-20T22:30:59.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:59.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:30:59.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[28216]: Adjusting osd_memory_target on smithi115 to 16064M 2024-09-20T22:30:59.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:59.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:59.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:30:59 smithi115 ceph-mon[28216]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:30:59.691 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:30:59.709 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi115:vg_nvme/lv_3 2024-09-20T22:30:59.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:59 smithi153 ceph-mon[24571]: Detected new or changed devices on smithi115 2024-09-20T22:30:59.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:59 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:59.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:59 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:30:59.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:59 smithi153 ceph-mon[24571]: Adjusting osd_memory_target on smithi115 to 16064M 2024-09-20T22:30:59.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:59 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:59.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:59 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:30:59.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:30:59 smithi153 ceph-mon[24571]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:01.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:01 smithi153 ceph-mon[24571]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:01.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:01 smithi115 ceph-mon[23720]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:01.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:01 smithi115 ceph-mon[28216]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:02.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:02 smithi115 ceph-mon[23720]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:02.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:02 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:31:02.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:02 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:31:02.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:02 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:02.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:02 smithi115 ceph-mon[28216]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:02.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:02 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:31:02.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:02 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:31:02.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:02 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:02.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:02 smithi153 ceph-mon[24571]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:02.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:02 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:31:02.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:02 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:31:02.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:02 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:03.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:03 smithi153 ceph-mon[24571]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:03.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:03 smithi115 ceph-mon[23720]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:03.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:03 smithi115 ceph-mon[28216]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:04.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:04 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3349631399' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7592a9d-d11a-48c3-9cc9-2ddb6dcb2ac9"}]: dispatch 2024-09-20T22:31:04.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:04 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3349631399' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7592a9d-d11a-48c3-9cc9-2ddb6dcb2ac9"}]': finished 2024-09-20T22:31:04.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:04 smithi115 ceph-mon[23720]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T22:31:04.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:04 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:04.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:04 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3937154538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:31:04.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:04 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3349631399' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7592a9d-d11a-48c3-9cc9-2ddb6dcb2ac9"}]: dispatch 2024-09-20T22:31:04.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:04 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3349631399' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7592a9d-d11a-48c3-9cc9-2ddb6dcb2ac9"}]': finished 2024-09-20T22:31:04.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:04 smithi115 ceph-mon[28216]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T22:31:04.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:04 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:04.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:04 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3937154538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:31:05.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:04 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3349631399' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7592a9d-d11a-48c3-9cc9-2ddb6dcb2ac9"}]: dispatch 2024-09-20T22:31:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:04 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3349631399' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7592a9d-d11a-48c3-9cc9-2ddb6dcb2ac9"}]': finished 2024-09-20T22:31:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:04 smithi153 ceph-mon[24571]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T22:31:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:04 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:04 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3937154538' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:31:06.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:05 smithi153 ceph-mon[24571]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:06.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:05 smithi115 ceph-mon[23720]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:06.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:05 smithi115 ceph-mon[28216]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:07.820 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:07 smithi115 ceph-mon[23720]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:07.820 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:07 smithi115 ceph-mon[28216]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:08.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:07 smithi153 ceph-mon[24571]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:09.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:09 smithi115 ceph-mon[23720]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:09.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:09 smithi115 ceph-mon[28216]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:10.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:09 smithi153 ceph-mon[24571]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:11.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:10 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T22:31:11.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:10 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:11.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:10 smithi153 ceph-mon[24571]: Deploying daemon osd.1 on smithi115 2024-09-20T22:31:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:10 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T22:31:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:10 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:10 smithi115 ceph-mon[23720]: Deploying daemon osd.1 on smithi115 2024-09-20T22:31:11.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:10 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T22:31:11.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:10 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:11.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:10 smithi115 ceph-mon[28216]: Deploying daemon osd.1 on smithi115 2024-09-20T22:31:11.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:11 smithi115 ceph-mon[23720]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:11.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:11 smithi115 ceph-mon[28216]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:12.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:11 smithi153 ceph-mon[24571]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:14.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:13 smithi153 ceph-mon[24571]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:13 smithi115 ceph-mon[23720]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:14.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:13 smithi115 ceph-mon[28216]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:16.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:16 smithi115 ceph-mon[28216]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:16.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:16 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:16.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:16 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:31:16.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:16 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:16.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:16 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:31:16.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:16 smithi115 ceph-mon[23720]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:16.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:16 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:16.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:16 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:31:16.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:16 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:16.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:16 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:31:16.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:16 smithi153 ceph-mon[24571]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:16.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:16 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:16.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:16 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:31:16.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:16 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:16.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:16 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:31:18.193 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:18 smithi115 ceph-mon[23720]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:18.193 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:18 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:18.194 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:18 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:18.194 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:18 smithi115 ceph-mon[28216]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:18.194 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:18 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:18.194 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:18 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:18.221 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 1 on host 'smithi115' 2024-09-20T22:31:18.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:18 smithi153 ceph-mon[24571]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:18.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:18 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:18.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:18 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:18.843 DEBUG:teuthology.orchestra.run.smithi115:osd.1> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.1.service 2024-09-20T22:31:18.845 INFO:tasks.cephadm:Deploying osd.2 on smithi115 with /dev/vg_nvme/lv_2... 2024-09-20T22:31:18.846 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T22:31:19.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:19 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:19.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:19 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:31:19.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:19 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:19.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:19 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:31:19.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:19 smithi153 ceph-mon[24571]: from='osd.1 [v2:172.21.15.115:6810/650753341,v1:172.21.15.115:6811/650753341]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:31:19.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:19 smithi153 ceph-mon[24571]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:31:19.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:19 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:19.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:19 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:31:19.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:19 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:19.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:19 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:31:19.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:19 smithi115 ceph-mon[23720]: from='osd.1 [v2:172.21.15.115:6810/650753341,v1:172.21.15.115:6811/650753341]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:31:19.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:19 smithi115 ceph-mon[23720]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:31:19.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:19 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:19.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:19 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:31:19.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:19 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:19.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:19 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:31:19.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:19 smithi115 ceph-mon[28216]: from='osd.1 [v2:172.21.15.115:6810/650753341,v1:172.21.15.115:6811/650753341]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:31:19.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:19 smithi115 ceph-mon[28216]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:31:20.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:20 smithi153 ceph-mon[24571]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:20.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:20 smithi153 ceph-mon[24571]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T22:31:20.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:20 smithi153 ceph-mon[24571]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T22:31:20.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:20 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:20.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:20 smithi153 ceph-mon[24571]: from='osd.1 [v2:172.21.15.115:6810/650753341,v1:172.21.15.115:6811/650753341]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:31:20.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:20 smithi153 ceph-mon[24571]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:31:20.590 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-mon[23720]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:20.590 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-mon[23720]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T22:31:20.590 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-mon[23720]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T22:31:20.590 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:20.590 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-mon[23720]: from='osd.1 [v2:172.21.15.115:6810/650753341,v1:172.21.15.115:6811/650753341]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:31:20.590 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-mon[23720]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:31:20.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-mon[28216]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:20.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-mon[28216]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T22:31:20.590 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-mon[28216]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T22:31:20.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:20.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-mon[28216]: from='osd.1 [v2:172.21.15.115:6810/650753341,v1:172.21.15.115:6811/650753341]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:31:20.591 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-mon[28216]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:31:20.591 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1[33983]: 2024-09-20T22:31:20.232+0000 7f3e4394f700 -1 osd.1 0 waiting for initial osdmap 2024-09-20T22:31:20.591 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:31:20 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1[33983]: 2024-09-20T22:31:20.235+0000 7f3e3e2e6700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:31:21.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:21 smithi153 ceph-mon[24571]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-09-20T22:31:21.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:21 smithi153 ceph-mon[24571]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T22:31:21.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:21.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:21 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:21.575 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:21 smithi115 ceph-mon[23720]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-09-20T22:31:21.575 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:21 smithi115 ceph-mon[23720]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T22:31:21.575 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:21.575 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:21 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:21.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:21 smithi115 ceph-mon[28216]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-09-20T22:31:21.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:21 smithi115 ceph-mon[28216]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T22:31:21.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:21.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:21 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:22.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[23720]: purged_snaps scrub starts 2024-09-20T22:31:22.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[23720]: purged_snaps scrub ok 2024-09-20T22:31:22.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[23720]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:22.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:22.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[23720]: osd.1 [v2:172.21.15.115:6810/650753341,v1:172.21.15.115:6811/650753341] boot 2024-09-20T22:31:22.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[23720]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T22:31:22.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:22.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:22.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:22.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:22.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:22.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:22.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[28216]: purged_snaps scrub starts 2024-09-20T22:31:22.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[28216]: purged_snaps scrub ok 2024-09-20T22:31:22.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[28216]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:22.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:22.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[28216]: osd.1 [v2:172.21.15.115:6810/650753341,v1:172.21.15.115:6811/650753341] boot 2024-09-20T22:31:22.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[28216]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T22:31:22.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:22.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:22.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:22.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:22.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:22.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:22 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:22.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:22 smithi153 ceph-mon[24571]: purged_snaps scrub starts 2024-09-20T22:31:22.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:22 smithi153 ceph-mon[24571]: purged_snaps scrub ok 2024-09-20T22:31:22.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:22 smithi153 ceph-mon[24571]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:31:22.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:22 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:22.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:22 smithi153 ceph-mon[24571]: osd.1 [v2:172.21.15.115:6810/650753341,v1:172.21.15.115:6811/650753341] boot 2024-09-20T22:31:22.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:22 smithi153 ceph-mon[24571]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T22:31:22.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:22 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:31:22.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:22 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:22.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:22 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:22.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:22 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:22.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:22 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:22.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:22 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:23.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:23 smithi153 ceph-mon[24571]: Detected new or changed devices on smithi115 2024-09-20T22:31:23.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:23 smithi153 ceph-mon[24571]: Adjusting osd_memory_target on smithi115 to 8032M 2024-09-20T22:31:23.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:23 smithi153 ceph-mon[24571]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T22:31:23.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:23 smithi115 ceph-mon[23720]: Detected new or changed devices on smithi115 2024-09-20T22:31:23.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:23 smithi115 ceph-mon[23720]: Adjusting osd_memory_target on smithi115 to 8032M 2024-09-20T22:31:23.530 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:23 smithi115 ceph-mon[23720]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T22:31:23.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:23 smithi115 ceph-mon[28216]: Detected new or changed devices on smithi115 2024-09-20T22:31:23.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:23 smithi115 ceph-mon[28216]: Adjusting osd_memory_target on smithi115 to 8032M 2024-09-20T22:31:23.531 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:23 smithi115 ceph-mon[28216]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T22:31:23.536 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:31:23.557 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi115:vg_nvme/lv_2 2024-09-20T22:31:24.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:24 smithi153 ceph-mon[24571]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:24.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:24 smithi115 ceph-mon[23720]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:24.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:24 smithi115 ceph-mon[28216]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:26.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:26 smithi153 ceph-mon[24571]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:26 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:31:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:26 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:31:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:26 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:26.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:26 smithi115 ceph-mon[23720]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:26.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:26 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:31:26.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:26 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:31:26.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:26 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:26.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:26 smithi115 ceph-mon[28216]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:26.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:26 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:31:26.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:26 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:31:26.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:26 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:27.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:27 smithi115 ceph-mon[23720]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:27.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:27 smithi115 ceph-mon[28216]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:27.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:27 smithi153 ceph-mon[24571]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:28.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:28 smithi115 ceph-mon[23720]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:28.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:28 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2757947824' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74152a6d-d11c-4405-b83f-248e8f225d0a"}]: dispatch 2024-09-20T22:31:28.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:28 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2757947824' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74152a6d-d11c-4405-b83f-248e8f225d0a"}]': finished 2024-09-20T22:31:28.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:28 smithi115 ceph-mon[23720]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T22:31:28.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:28 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:28.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:28 smithi115 ceph-mon[28216]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:28.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:28 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2757947824' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74152a6d-d11c-4405-b83f-248e8f225d0a"}]: dispatch 2024-09-20T22:31:28.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:28 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2757947824' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74152a6d-d11c-4405-b83f-248e8f225d0a"}]': finished 2024-09-20T22:31:28.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:28 smithi115 ceph-mon[28216]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T22:31:28.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:28 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:28.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:28 smithi153 ceph-mon[24571]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:28.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:28 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2757947824' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74152a6d-d11c-4405-b83f-248e8f225d0a"}]: dispatch 2024-09-20T22:31:28.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:28 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2757947824' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74152a6d-d11c-4405-b83f-248e8f225d0a"}]': finished 2024-09-20T22:31:28.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:28 smithi153 ceph-mon[24571]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T22:31:28.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:28 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:29.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:29 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2852617889' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:31:29.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:29 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2852617889' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:31:29.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:29 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2852617889' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:31:30.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:30 smithi153 ceph-mon[24571]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:30.634 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:30 smithi115 ceph-mon[23720]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:30.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:30 smithi115 ceph-mon[28216]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:32.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:32 smithi153 ceph-mon[24571]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:32.583 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:32 smithi115 ceph-mon[23720]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:32.583 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:32 smithi115 ceph-mon[28216]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:34.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:34 smithi115 ceph-mon[23720]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:34.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:34 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T22:31:34.500 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:34 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:34.500 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:34 smithi115 ceph-mon[28216]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:34.501 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:34 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T22:31:34.501 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:34 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:34.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:34 smithi153 ceph-mon[24571]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:34.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:34 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T22:31:34.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:34 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:35.467 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:35 smithi115 ceph-mon[23720]: Deploying daemon osd.2 on smithi115 2024-09-20T22:31:35.468 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:35 smithi115 ceph-mon[28216]: Deploying daemon osd.2 on smithi115 2024-09-20T22:31:35.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:35 smithi153 ceph-mon[24571]: Deploying daemon osd.2 on smithi115 2024-09-20T22:31:36.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:36 smithi153 ceph-mon[24571]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:36.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:36 smithi115 ceph-mon[23720]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:36.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:36 smithi115 ceph-mon[28216]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:37.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:31:37.300 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:37 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:31:37.300 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:37 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:31:37.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:37 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:31:37.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:37 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:31:37.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:37 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:31:38.374 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:38 smithi115 ceph-mon[23720]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:38.375 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:38 smithi115 ceph-mon[28216]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:38.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:38 smithi153 ceph-mon[24571]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:40.275 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:39 smithi115 ceph-mon[23720]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:40.275 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:39 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:40.276 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:39 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:31:40.276 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:39 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:40.276 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:39 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:31:40.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:39 smithi115 ceph-mon[28216]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:40.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:39 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:40.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:39 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:31:40.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:39 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:40.276 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:39 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:31:40.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:39 smithi153 ceph-mon[24571]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:40.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:39 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:40.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:39 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:31:40.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:39 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:39 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:31:42.003 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 2 on host 'smithi115' 2024-09-20T22:31:42.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:41 smithi153 ceph-mon[24571]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:42.338 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:41 smithi115 ceph-mon[23720]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:42.339 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:41 smithi115 ceph-mon[28216]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:42.611 DEBUG:teuthology.orchestra.run.smithi115:osd.2> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.2.service 2024-09-20T22:31:42.614 INFO:tasks.cephadm:Deploying osd.3 on smithi115 with /dev/vg_nvme/lv_1... 2024-09-20T22:31:42.614 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T22:31:43.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:43 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:43.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:43 smithi153 ceph-mon[24571]: from='osd.2 [v2:172.21.15.115:6818/123310734,v1:172.21.15.115:6819/123310734]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:31:43.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:43 smithi153 ceph-mon[24571]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:31:43.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:43 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:43.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:43 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:43.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:43 smithi115 ceph-mon[23720]: from='osd.2 [v2:172.21.15.115:6818/123310734,v1:172.21.15.115:6819/123310734]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:31:43.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:43 smithi115 ceph-mon[23720]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:31:43.292 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:43 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:43 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:43 smithi115 ceph-mon[28216]: from='osd.2 [v2:172.21.15.115:6818/123310734,v1:172.21.15.115:6819/123310734]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:31:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:43 smithi115 ceph-mon[28216]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:31:43.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:43 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:44.232 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-mon[23720]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:44.232 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-mon[23720]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T22:31:44.232 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-mon[23720]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T22:31:44.232 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:44.232 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-mon[23720]: from='osd.2 [v2:172.21.15.115:6818/123310734,v1:172.21.15.115:6819/123310734]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:31:44.232 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-mon[23720]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:31:44.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-mon[28216]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:44.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-mon[28216]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T22:31:44.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-mon[28216]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T22:31:44.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:44.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-mon[28216]: from='osd.2 [v2:172.21.15.115:6818/123310734,v1:172.21.15.115:6819/123310734]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:31:44.233 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-mon[28216]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:31:44.233 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2[36880]: 2024-09-20T22:31:44.013+0000 7f258e6bd700 -1 osd.2 0 waiting for initial osdmap 2024-09-20T22:31:44.234 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:31:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2[36880]: 2024-09-20T22:31:44.016+0000 7f2587050700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:31:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:44 smithi153 ceph-mon[24571]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:44 smithi153 ceph-mon[24571]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T22:31:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:44 smithi153 ceph-mon[24571]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T22:31:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:44 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:44 smithi153 ceph-mon[24571]: from='osd.2 [v2:172.21.15.115:6818/123310734,v1:172.21.15.115:6819/123310734]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:31:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:44 smithi153 ceph-mon[24571]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:31:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:45 smithi115 ceph-mon[23720]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-09-20T22:31:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:45 smithi115 ceph-mon[23720]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T22:31:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:45.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:45 smithi115 ceph-mon[28216]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-09-20T22:31:45.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:45 smithi115 ceph-mon[28216]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T22:31:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:45.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:45 smithi153 ceph-mon[24571]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-09-20T22:31:45.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:45 smithi153 ceph-mon[24571]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T22:31:45.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:45.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:46.074 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[23720]: purged_snaps scrub starts 2024-09-20T22:31:46.074 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[23720]: purged_snaps scrub ok 2024-09-20T22:31:46.074 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[23720]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:46.074 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:46.074 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[23720]: osd.2 [v2:172.21.15.115:6818/123310734,v1:172.21.15.115:6819/123310734] boot 2024-09-20T22:31:46.074 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[23720]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T22:31:46.075 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:46.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[28216]: purged_snaps scrub starts 2024-09-20T22:31:46.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[28216]: purged_snaps scrub ok 2024-09-20T22:31:46.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[28216]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:46.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:46.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[28216]: osd.2 [v2:172.21.15.115:6818/123310734,v1:172.21.15.115:6819/123310734] boot 2024-09-20T22:31:46.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[28216]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T22:31:46.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:46 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:46.079 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:31:46.101 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi115:vg_nvme/lv_1 2024-09-20T22:31:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:46 smithi153 ceph-mon[24571]: purged_snaps scrub starts 2024-09-20T22:31:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:46 smithi153 ceph-mon[24571]: purged_snaps scrub ok 2024-09-20T22:31:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:46 smithi153 ceph-mon[24571]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:31:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:46 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:46 smithi153 ceph-mon[24571]: osd.2 [v2:172.21.15.115:6818/123310734,v1:172.21.15.115:6819/123310734] boot 2024-09-20T22:31:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:46 smithi153 ceph-mon[24571]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T22:31:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:46 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:31:47.238 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:47 smithi115 ceph-mon[23720]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T22:31:47.239 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:47 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-20T22:31:47.239 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:47 smithi115 ceph-mon[28216]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T22:31:47.239 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:47 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-20T22:31:47.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:47 smithi153 ceph-mon[24571]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T22:31:47.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:47 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-20T22:31:48.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:48.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-20T22:31:48.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T22:31:48.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T22:31:48.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:48.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:48.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:48.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:48.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:48.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:48.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:31:48.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:48.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:31:48.129 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:48.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:48.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-20T22:31:48.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T22:31:48.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T22:31:48.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:48.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:48.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:48.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:48.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:48.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:48.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:31:48.131 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:48.131 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:31:48.131 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:48 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:48.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-20T22:31:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T22:31:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T22:31:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:31:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:31:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:31:48.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:48 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:49.127 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:31:48 smithi115 sudo[38216]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T22:31:49.127 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:31:48 smithi115 sudo[38216]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T22:31:49.127 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:31:48 smithi115 sudo[38216]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:31:49.127 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[23720]: Detected new or changed devices on smithi115 2024-09-20T22:31:49.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[23720]: Adjusting osd_memory_target on smithi115 to 5354M 2024-09-20T22:31:49.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T22:31:49.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[23720]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T22:31:49.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:31:49.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:31:49.128 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:49.128 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[28216]: Detected new or changed devices on smithi115 2024-09-20T22:31:49.128 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[28216]: Adjusting osd_memory_target on smithi115 to 5354M 2024-09-20T22:31:49.128 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T22:31:49.128 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[28216]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T22:31:49.128 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:31:49.128 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:31:49.129 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:49 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:49.401 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:31:49 smithi115 sudo[38295]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T22:31:49.401 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:31:49 smithi115 sudo[38295]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T22:31:49.401 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:31:49 smithi115 sudo[38295]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:31:49.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:49 smithi153 ceph-mon[24571]: Detected new or changed devices on smithi115 2024-09-20T22:31:49.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:49 smithi153 ceph-mon[24571]: Adjusting osd_memory_target on smithi115 to 5354M 2024-09-20T22:31:49.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:49 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T22:31:49.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:49 smithi153 ceph-mon[24571]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T22:31:49.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:49 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:31:49.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:49 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:31:49.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:49 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:49.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:31:49 smithi115 sudo[38303]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T22:31:49.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:31:49 smithi115 sudo[38303]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T22:31:49.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:31:49 smithi115 sudo[38303]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:31:49.919 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:31:49 smithi115 sudo[38318]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T22:31:49.919 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:31:49 smithi115 sudo[38318]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T22:31:49.919 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:31:49 smithi115 sudo[38318]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:31:49.919 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:31:49 smithi115 sudo[38325]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T22:31:49.919 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:31:49 smithi115 sudo[38325]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T22:31:50.171 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:50 smithi115 ceph-mon[23720]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:50.171 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:50 smithi115 ceph-mon[23720]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:50.171 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:50 smithi115 ceph-mon[23720]: osdmap e22: 3 total, 3 up, 3 in 2024-09-20T22:31:50.171 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:50 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:50.171 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:31:49 smithi115 sudo[38325]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:31:50.171 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:31:50 smithi115 sudo[38328]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T22:31:50.171 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:31:50 smithi115 sudo[38328]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T22:31:50.171 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:31:50 smithi115 sudo[38328]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:31:50.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:50 smithi115 ceph-mon[28216]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:50.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:50 smithi115 ceph-mon[28216]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:50.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:50 smithi115 ceph-mon[28216]: osdmap e22: 3 total, 3 up, 3 in 2024-09-20T22:31:50.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:50 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:50.430 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:50 smithi115 sudo[38355]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T22:31:50.430 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:50 smithi115 sudo[38355]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T22:31:50.431 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:50 smithi115 sudo[38355]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:31:50.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:50 smithi153 ceph-mon[24571]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:50.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:50 smithi153 ceph-mon[24571]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:31:50.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:50 smithi153 ceph-mon[24571]: osdmap e22: 3 total, 3 up, 3 in 2024-09-20T22:31:50.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:50 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:50.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:50 smithi115 sudo[38360]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T22:31:50.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:50 smithi115 sudo[38360]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T22:31:50.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:50 smithi115 sudo[38360]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:31:51.191 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:50 smithi115 sudo[38376]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T22:31:51.191 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:50 smithi115 sudo[38376]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T22:31:51.191 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:50 smithi115 sudo[38376]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:31:51.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:51 smithi115 sudo[38400]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T22:31:51.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:51 smithi115 sudo[38400]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T22:31:51.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:51 smithi115 sudo[38400]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:31:51.685 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 sudo[26081]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T22:31:51.685 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 sudo[26081]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T22:31:51.686 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 sudo[26081]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:31:51.686 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 sudo[26084]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T22:31:52.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 sudo[26084]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T22:31:52.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 sudo[26084]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:31:52.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 ceph-mon[24571]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:31:52.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 ceph-mon[24571]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:31:52.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:31:52.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:31:52.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:31:52.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:31:52.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:31:52.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:31:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[23720]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:31:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[23720]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:31:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:31:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:31:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:31:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:31:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:31:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:31:52.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[28216]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:31:52.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[28216]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:31:52.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:31:52.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:31:52.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:31:52.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:31:52.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:31:52.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:31:53.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:31:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1496292272' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ad34028-3e41-41c8-a7e5-4ff479f4287b"}]: dispatch 2024-09-20T22:31:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:31:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:31:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:31:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:31:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:31:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:31:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T22:31:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1496292272' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ad34028-3e41-41c8-a7e5-4ff479f4287b"}]': finished 2024-09-20T22:31:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T22:31:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:31:53.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:53.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:52 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/4259862342' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:31:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:31:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1496292272' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ad34028-3e41-41c8-a7e5-4ff479f4287b"}]: dispatch 2024-09-20T22:31:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:31:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:31:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:31:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:31:53.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:31:53.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:31:53.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T22:31:53.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1496292272' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ad34028-3e41-41c8-a7e5-4ff479f4287b"}]': finished 2024-09-20T22:31:53.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T22:31:53.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:31:53.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:53.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/4259862342' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:31:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:31:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1496292272' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ad34028-3e41-41c8-a7e5-4ff479f4287b"}]: dispatch 2024-09-20T22:31:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:31:53.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:31:53.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:31:53.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:31:53.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:31:53.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:31:53.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T22:31:53.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1496292272' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ad34028-3e41-41c8-a7e5-4ff479f4287b"}]': finished 2024-09-20T22:31:53.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T22:31:53.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:31:53.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:31:53.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:52 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/4259862342' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:31:54.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:53 smithi153 ceph-mon[24571]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:53 smithi115 ceph-mon[23720]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:54.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:53 smithi115 ceph-mon[28216]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:56.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:55 smithi153 ceph-mon[24571]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:56.134 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:55 smithi115 ceph-mon[23720]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:56.135 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:55 smithi115 ceph-mon[28216]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:57.842 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:57 smithi115 ceph-mon[23720]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:57.842 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:57 smithi115 ceph-mon[28216]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:58.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:57 smithi153 ceph-mon[24571]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:58.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:58 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T22:31:58.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:58 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:58.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:58 smithi115 ceph-mon[23720]: Deploying daemon osd.3 on smithi115 2024-09-20T22:31:58.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:58 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T22:31:58.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:58 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:58.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:58 smithi115 ceph-mon[28216]: Deploying daemon osd.3 on smithi115 2024-09-20T22:31:59.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:58 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T22:31:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:58 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:31:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:58 smithi153 ceph-mon[24571]: Deploying daemon osd.3 on smithi115 2024-09-20T22:31:59.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:31:59 smithi115 ceph-mon[23720]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:31:59.890 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:31:59 smithi115 ceph-mon[28216]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:00.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:31:59 smithi153 ceph-mon[24571]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:02.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:01 smithi153 ceph-mon[24571]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:02.122 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:01 smithi115 ceph-mon[23720]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:02.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:01 smithi115 ceph-mon[28216]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:04.240 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:03 smithi115 ceph-mon[23720]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:04.240 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:03 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:04.240 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:03 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:04.240 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:03 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:04.240 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:03 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:04.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:03 smithi115 ceph-mon[28216]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:04.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:03 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:04.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:03 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:04.240 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:03 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:04.241 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:03 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:04.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:03 smithi153 ceph-mon[24571]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:04.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:03 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:04.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:03 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:04.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:03 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:04.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:03 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:06.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:05 smithi115 ceph-mon[23720]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:06.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:05 smithi115 ceph-mon[28216]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:06.187 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 3 on host 'smithi115' 2024-09-20T22:32:06.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:05 smithi153 ceph-mon[24571]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:06.768 DEBUG:teuthology.orchestra.run.smithi115:osd.3> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.3.service 2024-09-20T22:32:06.770 INFO:tasks.cephadm:Deploying osd.4 on smithi153 with /dev/vg_nvme/lv_4... 2024-09-20T22:32:06.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T22:32:07.023 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:06 smithi115 ceph-mon[23720]: from='osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:32:07.023 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:06 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:07.023 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:06 smithi115 ceph-mon[28216]: from='osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:32:07.023 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:06 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:07.081 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:06 smithi153 ceph-mon[24571]: from='osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:32:07.081 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:06 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:08.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:07 smithi153 ceph-mon[24571]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:08.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:07 smithi153 ceph-mon[24571]: from='osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T22:32:08.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:07 smithi153 ceph-mon[24571]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T22:32:08.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:07 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:08.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:07 smithi153 ceph-mon[24571]: from='osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:32:08.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:07 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:08.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-mon[23720]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:08.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-mon[23720]: from='osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T22:32:08.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-mon[23720]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T22:32:08.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:08.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-mon[23720]: from='osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:32:08.302 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:08.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-mon[28216]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:08.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-mon[28216]: from='osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T22:32:08.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-mon[28216]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T22:32:08.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:08.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-mon[28216]: from='osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:32:08.303 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:08.303 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3[39848]: 2024-09-20T22:32:07.972+0000 7f3c9a3e8700 -1 osd.3 0 waiting for initial osdmap 2024-09-20T22:32:08.303 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:32:07 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3[39848]: 2024-09-20T22:32:07.978+0000 7f3c96d83700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:32:09.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:08 smithi153 ceph-mon[24571]: from='osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-09-20T22:32:09.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:08 smithi153 ceph-mon[24571]: osdmap e25: 4 total, 3 up, 4 in 2024-09-20T22:32:09.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:08 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:09.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:08 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:09.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:08 smithi115 ceph-mon[23720]: from='osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-09-20T22:32:09.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:08 smithi115 ceph-mon[23720]: osdmap e25: 4 total, 3 up, 4 in 2024-09-20T22:32:09.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:08 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:09.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:08 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:09.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:08 smithi115 ceph-mon[28216]: from='osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-09-20T22:32:09.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:08 smithi115 ceph-mon[28216]: osdmap e25: 4 total, 3 up, 4 in 2024-09-20T22:32:09.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:08 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:09.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:08 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:32:09.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-09-20T22:32:10.166 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[23720]: purged_snaps scrub starts 2024-09-20T22:32:10.166 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[23720]: purged_snaps scrub ok 2024-09-20T22:32:10.166 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[23720]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:10.166 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:10.166 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[23720]: osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919] boot 2024-09-20T22:32:10.166 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[23720]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T22:32:10.166 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:10.167 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[28216]: purged_snaps scrub starts 2024-09-20T22:32:10.167 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[28216]: purged_snaps scrub ok 2024-09-20T22:32:10.167 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[28216]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:10.167 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:10.167 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[28216]: osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919] boot 2024-09-20T22:32:10.167 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[28216]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T22:32:10.167 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:09 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:10.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:09 smithi153 ceph-mon[24571]: purged_snaps scrub starts 2024-09-20T22:32:10.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:09 smithi153 ceph-mon[24571]: purged_snaps scrub ok 2024-09-20T22:32:10.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:09 smithi153 ceph-mon[24571]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:32:10.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:09 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:10.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:09 smithi153 ceph-mon[24571]: osd.3 [v2:172.21.15.115:6826/1168172919,v1:172.21.15.115:6827/1168172919] boot 2024-09-20T22:32:10.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:09 smithi153 ceph-mon[24571]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T22:32:10.214 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:09 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:32:11.118 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: osdmap e27: 4 total, 4 up, 4 in 2024-09-20T22:32:11.118 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: Detected new or changed devices on smithi115 2024-09-20T22:32:11.118 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:11.118 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:11.118 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:11.118 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:11.118 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:11.118 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: Adjusting osd_memory_target on smithi115 to 4016M 2024-09-20T22:32:11.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:11.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:11.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:11.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:11.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:11.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:10 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:11.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: osdmap e27: 4 total, 4 up, 4 in 2024-09-20T22:32:11.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: Detected new or changed devices on smithi115 2024-09-20T22:32:11.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:11.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:11.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:11.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:11.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:11.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: Adjusting osd_memory_target on smithi115 to 4016M 2024-09-20T22:32:11.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:11.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:11.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:11.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:11.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:11.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:11.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: osdmap e27: 4 total, 4 up, 4 in 2024-09-20T22:32:11.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: Detected new or changed devices on smithi115 2024-09-20T22:32:11.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:11.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:11.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:11.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:11.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:11.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: Adjusting osd_memory_target on smithi115 to 4016M 2024-09-20T22:32:11.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:11.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:11.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:11.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:11.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:11.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:10 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:12.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:12 smithi153 ceph-mon[24571]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:12.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:12 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:32:12.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:12 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:32:12.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:12 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:12.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:12 smithi115 ceph-mon[23720]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:12.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:12 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:32:12.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:12 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:32:12.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:12 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:12.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:12 smithi115 ceph-mon[28216]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:12.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:12 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:32:12.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:12 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:32:12.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:12 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:13.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:13 smithi153 ceph-mon[24571]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:13.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:13 smithi115 ceph-mon[23720]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:13.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:13 smithi115 ceph-mon[28216]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:14.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:14 smithi153 ceph-mon[24571]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:14.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:14 smithi153 ceph-mon[24571]: from='client.? 172.21.15.153:0/3457215204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bffa26fb-977a-41d0-a298-8671fdb777e4"}]: dispatch 2024-09-20T22:32:14.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:14 smithi153 ceph-mon[24571]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bffa26fb-977a-41d0-a298-8671fdb777e4"}]: dispatch 2024-09-20T22:32:14.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:14 smithi153 ceph-mon[24571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bffa26fb-977a-41d0-a298-8671fdb777e4"}]': finished 2024-09-20T22:32:14.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:14 smithi153 ceph-mon[24571]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T22:32:14.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:14 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:14.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:14 smithi115 ceph-mon[23720]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:14.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:14 smithi115 ceph-mon[23720]: from='client.? 172.21.15.153:0/3457215204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bffa26fb-977a-41d0-a298-8671fdb777e4"}]: dispatch 2024-09-20T22:32:14.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:14 smithi115 ceph-mon[23720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bffa26fb-977a-41d0-a298-8671fdb777e4"}]: dispatch 2024-09-20T22:32:14.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:14 smithi115 ceph-mon[23720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bffa26fb-977a-41d0-a298-8671fdb777e4"}]': finished 2024-09-20T22:32:14.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:14 smithi115 ceph-mon[23720]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T22:32:14.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:14 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:14.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:14 smithi115 ceph-mon[28216]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:14.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:14 smithi115 ceph-mon[28216]: from='client.? 172.21.15.153:0/3457215204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bffa26fb-977a-41d0-a298-8671fdb777e4"}]: dispatch 2024-09-20T22:32:14.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:14 smithi115 ceph-mon[28216]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bffa26fb-977a-41d0-a298-8671fdb777e4"}]: dispatch 2024-09-20T22:32:14.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:14 smithi115 ceph-mon[28216]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bffa26fb-977a-41d0-a298-8671fdb777e4"}]': finished 2024-09-20T22:32:14.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:14 smithi115 ceph-mon[28216]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T22:32:14.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:14 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:15.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:15 smithi115 ceph-mon[23720]: from='client.? 172.21.15.153:0/2422583368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:32:15.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:15 smithi115 ceph-mon[28216]: from='client.? 172.21.15.153:0/2422583368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:32:15.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:15 smithi153 ceph-mon[24571]: from='client.? 172.21.15.153:0/2422583368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:32:16.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:16 smithi153 ceph-mon[24571]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:16.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:16 smithi115 ceph-mon[23720]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:16.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:16 smithi115 ceph-mon[28216]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:18.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:18 smithi153 ceph-mon[24571]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:18.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:18 smithi115 ceph-mon[23720]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:18.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:18 smithi115 ceph-mon[28216]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:20.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:20 smithi153 ceph-mon[24571]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:20.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:20 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T22:32:20.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:20 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:20.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:20 smithi115 ceph-mon[23720]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:20.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:20 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T22:32:20.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:20 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:20.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:20 smithi115 ceph-mon[28216]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:20.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:20 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T22:32:20.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:20 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:21.471 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:21 smithi153 ceph-mon[24571]: Deploying daemon osd.4 on smithi153 2024-09-20T22:32:21.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:21 smithi115 ceph-mon[23720]: Deploying daemon osd.4 on smithi153 2024-09-20T22:32:21.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:21 smithi115 ceph-mon[28216]: Deploying daemon osd.4 on smithi153 2024-09-20T22:32:22.491 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:22 smithi153 ceph-mon[24571]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:22.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:22 smithi115 ceph-mon[23720]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:22.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:22 smithi115 ceph-mon[28216]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:24.311 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:24 smithi153 ceph-mon[24571]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:24.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:24 smithi115 ceph-mon[23720]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:24.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:24 smithi115 ceph-mon[28216]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:25 smithi115 ceph-mon[23720]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:25 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:25 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:25 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:25 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:26.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:25 smithi115 ceph-mon[28216]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:26.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:25 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:26.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:25 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:26.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:25 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:26.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:25 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:26.184 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:25 smithi153 ceph-mon[24571]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:26.184 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:25 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:26.184 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:25 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:26.184 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:25 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:26.184 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:25 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:27.581 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 4 on host 'smithi153' 2024-09-20T22:32:28.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:27 smithi115 ceph-mon[23720]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:28.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:27 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:28.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:27 smithi115 ceph-mon[28216]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:28.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:27 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:28.189 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:27 smithi153 ceph-mon[24571]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:28.189 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:27 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:28.506 DEBUG:teuthology.orchestra.run.smithi153:osd.4> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.4.service 2024-09-20T22:32:28.508 INFO:tasks.cephadm:Deploying osd.5 on smithi153 with /dev/vg_nvme/lv_3... 2024-09-20T22:32:28.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T22:32:29.109 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:28 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:29.109 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:28 smithi153 ceph-mon[24571]: from='osd.4 [v2:172.21.15.153:6800/46581367,v1:172.21.15.153:6801/46581367]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:32:29.110 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:28 smithi153 ceph-mon[24571]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:32:29.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:28 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:29.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:28 smithi115 ceph-mon[23720]: from='osd.4 [v2:172.21.15.153:6800/46581367,v1:172.21.15.153:6801/46581367]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:32:29.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:28 smithi115 ceph-mon[23720]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:32:29.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:28 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:29.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:28 smithi115 ceph-mon[28216]: from='osd.4 [v2:172.21.15.153:6800/46581367,v1:172.21.15.153:6801/46581367]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:32:29.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:28 smithi115 ceph-mon[28216]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:32:30.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:29 smithi153 ceph-mon[24571]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:30.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:29 smithi153 ceph-mon[24571]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T22:32:30.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:29 smithi153 ceph-mon[24571]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T22:32:30.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:29 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:30.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:29 smithi153 ceph-mon[24571]: from='osd.4 [v2:172.21.15.153:6800/46581367,v1:172.21.15.153:6801/46581367]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:32:30.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:29 smithi153 ceph-mon[24571]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:32:30.120 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:32:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[27910]: 2024-09-20T22:32:29.964+0000 7fdf4e323700 -1 osd.4 0 waiting for initial osdmap 2024-09-20T22:32:30.120 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:32:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[27910]: 2024-09-20T22:32:29.968+0000 7fdf48cba700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:32:30.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:29 smithi115 ceph-mon[23720]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:30.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:29 smithi115 ceph-mon[23720]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T22:32:30.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:29 smithi115 ceph-mon[23720]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T22:32:30.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:29 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:30.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:29 smithi115 ceph-mon[23720]: from='osd.4 [v2:172.21.15.153:6800/46581367,v1:172.21.15.153:6801/46581367]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:32:30.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:29 smithi115 ceph-mon[23720]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:32:30.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:29 smithi115 ceph-mon[28216]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:30.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:29 smithi115 ceph-mon[28216]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T22:32:30.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:29 smithi115 ceph-mon[28216]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T22:32:30.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:29 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:30.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:29 smithi115 ceph-mon[28216]: from='osd.4 [v2:172.21.15.153:6800/46581367,v1:172.21.15.153:6801/46581367]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:32:30.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:29 smithi115 ceph-mon[28216]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:32:31.265 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:30 smithi153 ceph-mon[24571]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-09-20T22:32:31.265 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:30 smithi153 ceph-mon[24571]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T22:32:31.265 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:30 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:31.265 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:30 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:31.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:30 smithi115 ceph-mon[23720]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-09-20T22:32:31.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:30 smithi115 ceph-mon[23720]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T22:32:31.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:30 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:31.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:30 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:31.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:30 smithi115 ceph-mon[28216]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-09-20T22:32:31.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:30 smithi115 ceph-mon[28216]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T22:32:31.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:30 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:31.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:30 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:31.528 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:32:31.549 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-09-20T22:32:32.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:31 smithi153 ceph-mon[24571]: purged_snaps scrub starts 2024-09-20T22:32:32.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:31 smithi153 ceph-mon[24571]: purged_snaps scrub ok 2024-09-20T22:32:32.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:31 smithi153 ceph-mon[24571]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:32.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:31 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:32.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:31 smithi153 ceph-mon[24571]: osd.4 [v2:172.21.15.153:6800/46581367,v1:172.21.15.153:6801/46581367] boot 2024-09-20T22:32:32.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:31 smithi153 ceph-mon[24571]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T22:32:32.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:31 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:32.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[23720]: purged_snaps scrub starts 2024-09-20T22:32:32.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[23720]: purged_snaps scrub ok 2024-09-20T22:32:32.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[23720]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:32.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:32.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[23720]: osd.4 [v2:172.21.15.153:6800/46581367,v1:172.21.15.153:6801/46581367] boot 2024-09-20T22:32:32.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[23720]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T22:32:32.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:32.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[28216]: purged_snaps scrub starts 2024-09-20T22:32:32.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[28216]: purged_snaps scrub ok 2024-09-20T22:32:32.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[28216]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:32:32.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:32.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[28216]: osd.4 [v2:172.21.15.153:6800/46581367,v1:172.21.15.153:6801/46581367] boot 2024-09-20T22:32:32.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[28216]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T22:32:32.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:31 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:32:33.218 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:32 smithi153 ceph-mon[24571]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T22:32:33.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:32 smithi115 ceph-mon[23720]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T22:32:33.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:32 smithi115 ceph-mon[28216]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T22:32:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:33 smithi153 ceph-mon[24571]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:33 smithi153 ceph-mon[24571]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T22:32:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:33 smithi153 ceph-mon[24571]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T22:32:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:34.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[23720]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:34.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[23720]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T22:32:34.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[23720]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T22:32:34.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:34.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:34.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:34.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:34.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:34.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:34.306 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:34.307 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:34.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[28216]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:34.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[28216]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T22:32:34.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[28216]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T22:32:34.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:34.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:34.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:34.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:34.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:34.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:34.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:34.307 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:35.220 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:34 smithi153 ceph-mon[24571]: Detected new or changed devices on smithi153 2024-09-20T22:32:35.220 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:34 smithi153 ceph-mon[24571]: Adjusting osd_memory_target on smithi153 to 17088M 2024-09-20T22:32:35.220 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:34 smithi153 ceph-mon[24571]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T22:32:35.220 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:34 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:32:35.220 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:34 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:32:35.220 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:34 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:35.326 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:34 smithi115 ceph-mon[23720]: Detected new or changed devices on smithi153 2024-09-20T22:32:35.326 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:34 smithi115 ceph-mon[23720]: Adjusting osd_memory_target on smithi153 to 17088M 2024-09-20T22:32:35.327 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:34 smithi115 ceph-mon[23720]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T22:32:35.327 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:34 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:32:35.327 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:34 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:32:35.327 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:34 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:35.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:34 smithi115 ceph-mon[28216]: Detected new or changed devices on smithi153 2024-09-20T22:32:35.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:34 smithi115 ceph-mon[28216]: Adjusting osd_memory_target on smithi153 to 17088M 2024-09-20T22:32:35.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:34 smithi115 ceph-mon[28216]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T22:32:35.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:34 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:32:35.327 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:34 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:32:35.328 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:34 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:35 smithi115 ceph-mon[23720]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:35 smithi115 ceph-mon[23720]: from='client.34211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:36.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:35 smithi115 ceph-mon[28216]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:36.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:35 smithi115 ceph-mon[28216]: from='client.34211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:36.182 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:35 smithi153 ceph-mon[24571]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:36.182 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:35 smithi153 ceph-mon[24571]: from='client.34211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:37.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:36 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:32:37.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:36 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:32:37.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:36 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:32:37.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:36 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:32:37.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:37 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:32:37.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:37 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:32:38.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:38 smithi153 ceph-mon[24571]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:38.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:38 smithi153 ceph-mon[24571]: from='client.? 172.21.15.153:0/470000910' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cca50832-0a04-4e35-a109-dfe5da791cde"}]: dispatch 2024-09-20T22:32:38.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:38 smithi153 ceph-mon[24571]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cca50832-0a04-4e35-a109-dfe5da791cde"}]: dispatch 2024-09-20T22:32:38.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:38 smithi153 ceph-mon[24571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cca50832-0a04-4e35-a109-dfe5da791cde"}]': finished 2024-09-20T22:32:38.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:38 smithi153 ceph-mon[24571]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T22:32:38.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:38 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:38.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:38 smithi153 ceph-mon[24571]: from='client.? 172.21.15.153:0/1699571391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:32:38.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[23720]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:38.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[23720]: from='client.? 172.21.15.153:0/470000910' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cca50832-0a04-4e35-a109-dfe5da791cde"}]: dispatch 2024-09-20T22:32:38.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[23720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cca50832-0a04-4e35-a109-dfe5da791cde"}]: dispatch 2024-09-20T22:32:38.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[23720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cca50832-0a04-4e35-a109-dfe5da791cde"}]': finished 2024-09-20T22:32:38.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[23720]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T22:32:38.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:38.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[23720]: from='client.? 172.21.15.153:0/1699571391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:32:38.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[28216]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:38.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[28216]: from='client.? 172.21.15.153:0/470000910' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cca50832-0a04-4e35-a109-dfe5da791cde"}]: dispatch 2024-09-20T22:32:38.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[28216]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cca50832-0a04-4e35-a109-dfe5da791cde"}]: dispatch 2024-09-20T22:32:38.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[28216]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cca50832-0a04-4e35-a109-dfe5da791cde"}]': finished 2024-09-20T22:32:38.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[28216]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T22:32:38.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:38.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:38 smithi115 ceph-mon[28216]: from='client.? 172.21.15.153:0/1699571391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:32:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:40 smithi153 ceph-mon[24571]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-20T22:32:40.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:40 smithi115 ceph-mon[23720]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-20T22:32:40.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:40 smithi115 ceph-mon[28216]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-20T22:32:42.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:42 smithi153 ceph-mon[24571]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-20T22:32:42.334 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:42 smithi115 ceph-mon[23720]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-20T22:32:42.334 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:42 smithi115 ceph-mon[28216]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-20T22:32:44.072 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:44 smithi153 ceph-mon[24571]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T22:32:44.072 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:44 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T22:32:44.072 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:44 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:44.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:44 smithi115 ceph-mon[28216]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T22:32:44.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:44 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T22:32:44.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:44 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:44.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:44 smithi115 ceph-mon[23720]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T22:32:44.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:44 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T22:32:44.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:44 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:45.174 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:45 smithi153 ceph-mon[24571]: Deploying daemon osd.5 on smithi153 2024-09-20T22:32:45.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:45 smithi115 ceph-mon[23720]: Deploying daemon osd.5 on smithi153 2024-09-20T22:32:45.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:45 smithi115 ceph-mon[28216]: Deploying daemon osd.5 on smithi153 2024-09-20T22:32:46.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:46 smithi153 ceph-mon[24571]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-20T22:32:46.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:46 smithi115 ceph-mon[23720]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-20T22:32:46.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:46 smithi115 ceph-mon[28216]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-20T22:32:48.084 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:48 smithi153 ceph-mon[24571]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-20T22:32:48.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:48 smithi115 ceph-mon[23720]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-20T22:32:48.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:48 smithi115 ceph-mon[28216]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-20T22:32:50.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:49 smithi153 ceph-mon[24571]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 46 KiB/s, 0 objects/s recovering 2024-09-20T22:32:50.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:49 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:50.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:49 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:50.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:49 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:50.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:49 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:50.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:49 smithi115 ceph-mon[23720]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 46 KiB/s, 0 objects/s recovering 2024-09-20T22:32:50.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:49 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:50.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:49 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:50.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:49 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:50.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:49 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:49 smithi115 ceph-mon[28216]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 46 KiB/s, 0 objects/s recovering 2024-09-20T22:32:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:49 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:49 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:49 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:49 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:51.937 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:51 smithi153 ceph-mon[24571]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:51.968 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 5 on host 'smithi153' 2024-09-20T22:32:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:51 smithi115 ceph-mon[23720]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:52.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:51 smithi115 ceph-mon[28216]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:52.597 DEBUG:teuthology.orchestra.run.smithi153:osd.5> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.5.service 2024-09-20T22:32:52.599 INFO:tasks.cephadm:Deploying osd.6 on smithi153 with /dev/vg_nvme/lv_2... 2024-09-20T22:32:52.600 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T22:32:52.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:52 smithi153 ceph-mon[24571]: from='osd.5 [v2:172.21.15.153:6808/2463040153,v1:172.21.15.153:6809/2463040153]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:32:52.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:52 smithi153 ceph-mon[24571]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:32:52.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:52 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:52 smithi115 ceph-mon[23720]: from='osd.5 [v2:172.21.15.153:6808/2463040153,v1:172.21.15.153:6809/2463040153]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:32:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:52 smithi115 ceph-mon[23720]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:32:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:52 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:53.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:52 smithi115 ceph-mon[28216]: from='osd.5 [v2:172.21.15.153:6808/2463040153,v1:172.21.15.153:6809/2463040153]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:32:53.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:52 smithi115 ceph-mon[28216]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:32:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:52 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:53.958 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:53 smithi153 ceph-mon[24571]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:53.958 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:53 smithi153 ceph-mon[24571]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T22:32:53.958 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:53 smithi153 ceph-mon[24571]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T22:32:53.958 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:53 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:53.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:53 smithi153 ceph-mon[24571]: from='osd.5 [v2:172.21.15.153:6808/2463040153,v1:172.21.15.153:6809/2463040153]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:32:53.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:53 smithi153 ceph-mon[24571]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:32:53.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:53 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:53.959 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:32:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[30837]: 2024-09-20T22:32:53.719+0000 7fe2cd6c1700 -1 osd.5 0 waiting for initial osdmap 2024-09-20T22:32:53.959 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:32:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[30837]: 2024-09-20T22:32:53.724+0000 7fe2c6054700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:32:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[23720]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[23720]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T22:32:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[23720]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T22:32:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[23720]: from='osd.5 [v2:172.21.15.153:6808/2463040153,v1:172.21.15.153:6809/2463040153]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:32:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[23720]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:32:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:54.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[28216]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:54.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[28216]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T22:32:54.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[28216]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T22:32:54.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:54.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[28216]: from='osd.5 [v2:172.21.15.153:6808/2463040153,v1:172.21.15.153:6809/2463040153]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:32:54.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[28216]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:32:54.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:53 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:54.937 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:54 smithi153 ceph-mon[24571]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-09-20T22:32:54.937 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:54 smithi153 ceph-mon[24571]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T22:32:54.937 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:54 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:54.937 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:54 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:54 smithi115 ceph-mon[23720]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-09-20T22:32:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:54 smithi115 ceph-mon[23720]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T22:32:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:54 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:54 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:55.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:54 smithi115 ceph-mon[28216]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-09-20T22:32:55.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:54 smithi115 ceph-mon[28216]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T22:32:55.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:54 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:55.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:54 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:55.927 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:55 smithi153 ceph-mon[24571]: purged_snaps scrub starts 2024-09-20T22:32:55.927 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:55 smithi153 ceph-mon[24571]: purged_snaps scrub ok 2024-09-20T22:32:55.928 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:55 smithi153 ceph-mon[24571]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:55.928 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:55 smithi153 ceph-mon[24571]: osd.5 [v2:172.21.15.153:6808/2463040153,v1:172.21.15.153:6809/2463040153] boot 2024-09-20T22:32:55.928 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:55 smithi153 ceph-mon[24571]: osdmap e39: 6 total, 6 up, 6 in 2024-09-20T22:32:55.928 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:55 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:55.928 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:55 smithi153 ceph-mon[24571]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T22:32:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[23720]: purged_snaps scrub starts 2024-09-20T22:32:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[23720]: purged_snaps scrub ok 2024-09-20T22:32:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[23720]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[23720]: osd.5 [v2:172.21.15.153:6808/2463040153,v1:172.21.15.153:6809/2463040153] boot 2024-09-20T22:32:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[23720]: osdmap e39: 6 total, 6 up, 6 in 2024-09-20T22:32:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[23720]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T22:32:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[28216]: purged_snaps scrub starts 2024-09-20T22:32:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[28216]: purged_snaps scrub ok 2024-09-20T22:32:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[28216]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:32:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[28216]: osd.5 [v2:172.21.15.153:6808/2463040153,v1:172.21.15.153:6809/2463040153] boot 2024-09-20T22:32:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[28216]: osdmap e39: 6 total, 6 up, 6 in 2024-09-20T22:32:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:32:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:55 smithi115 ceph-mon[28216]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T22:32:56.217 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:32:56.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-09-20T22:32:58.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:57 smithi153 ceph-mon[24571]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:32:58.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:57 smithi153 ceph-mon[24571]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T22:32:58.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:57 smithi115 ceph-mon[23720]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:32:58.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:57 smithi115 ceph-mon[23720]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T22:32:58.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:57 smithi115 ceph-mon[28216]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:32:58.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:57 smithi115 ceph-mon[28216]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T22:32:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:58 smithi153 ceph-mon[24571]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T22:32:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:58 smithi153 ceph-mon[24571]: Detected new or changed devices on smithi153 2024-09-20T22:32:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:58 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:58 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:58 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:58 smithi153 ceph-mon[24571]: Adjusting osd_memory_target on smithi153 to 8544M 2024-09-20T22:32:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:58 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:58 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:58 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:59.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:58 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:59.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:58 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:59.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:58 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:59.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[23720]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T22:32:59.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[23720]: Detected new or changed devices on smithi153 2024-09-20T22:32:59.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:59.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:59.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:59.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[23720]: Adjusting osd_memory_target on smithi153 to 8544M 2024-09-20T22:32:59.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:59.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:59.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:59.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:59.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:59.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:59.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[28216]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T22:32:59.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[28216]: Detected new or changed devices on smithi153 2024-09-20T22:32:59.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:59.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:59.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:32:59.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[28216]: Adjusting osd_memory_target on smithi153 to 8544M 2024-09-20T22:32:59.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:59.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:59.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:32:59.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:32:59.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:32:59.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:58 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:32:59.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:59 smithi153 ceph-mon[24571]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:32:59.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:59 smithi153 ceph-mon[24571]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T22:32:59.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:59 smithi153 ceph-mon[24571]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:32:59.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:59 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:32:59.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:59 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:32:59.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:32:59 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:59 smithi115 ceph-mon[23720]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:59 smithi115 ceph-mon[23720]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T22:33:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:59 smithi115 ceph-mon[23720]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:59 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:33:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:59 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:33:00.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:32:59 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:59 smithi115 ceph-mon[28216]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:59 smithi115 ceph-mon[28216]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T22:33:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:59 smithi115 ceph-mon[28216]: from='client.34238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:59 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:33:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:59 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:33:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:32:59 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:01 smithi153 ceph-mon[24571]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:01 smithi153 ceph-mon[24571]: from='client.? 172.21.15.153:0/1699522840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60e07fca-cfbd-45d5-a409-94f5e90afd26"}]: dispatch 2024-09-20T22:33:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:01 smithi153 ceph-mon[24571]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60e07fca-cfbd-45d5-a409-94f5e90afd26"}]: dispatch 2024-09-20T22:33:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:01 smithi153 ceph-mon[24571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60e07fca-cfbd-45d5-a409-94f5e90afd26"}]': finished 2024-09-20T22:33:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:01 smithi153 ceph-mon[24571]: osdmap e44: 7 total, 6 up, 7 in 2024-09-20T22:33:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:01 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:02.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:01 smithi115 ceph-mon[23720]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:02.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:01 smithi115 ceph-mon[23720]: from='client.? 172.21.15.153:0/1699522840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60e07fca-cfbd-45d5-a409-94f5e90afd26"}]: dispatch 2024-09-20T22:33:02.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:01 smithi115 ceph-mon[23720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60e07fca-cfbd-45d5-a409-94f5e90afd26"}]: dispatch 2024-09-20T22:33:02.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:01 smithi115 ceph-mon[23720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60e07fca-cfbd-45d5-a409-94f5e90afd26"}]': finished 2024-09-20T22:33:02.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:01 smithi115 ceph-mon[23720]: osdmap e44: 7 total, 6 up, 7 in 2024-09-20T22:33:02.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:01 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:02.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:01 smithi115 ceph-mon[28216]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:02.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:01 smithi115 ceph-mon[28216]: from='client.? 172.21.15.153:0/1699522840' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60e07fca-cfbd-45d5-a409-94f5e90afd26"}]: dispatch 2024-09-20T22:33:02.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:01 smithi115 ceph-mon[28216]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60e07fca-cfbd-45d5-a409-94f5e90afd26"}]: dispatch 2024-09-20T22:33:02.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:01 smithi115 ceph-mon[28216]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60e07fca-cfbd-45d5-a409-94f5e90afd26"}]': finished 2024-09-20T22:33:02.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:01 smithi115 ceph-mon[28216]: osdmap e44: 7 total, 6 up, 7 in 2024-09-20T22:33:02.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:01 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:03.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:02 smithi153 ceph-mon[24571]: from='client.? 172.21.15.153:0/3322647570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:33:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:02 smithi115 ceph-mon[23720]: from='client.? 172.21.15.153:0/3322647570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:33:03.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:02 smithi115 ceph-mon[28216]: from='client.? 172.21.15.153:0/3322647570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:33:04.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:03 smithi153 ceph-mon[24571]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:04.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:03 smithi115 ceph-mon[23720]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:04.055 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:03 smithi115 ceph-mon[28216]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:06.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:05 smithi153 ceph-mon[24571]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:06.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:05 smithi115 ceph-mon[23720]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:06.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:05 smithi115 ceph-mon[28216]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:08.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:07 smithi153 ceph-mon[24571]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:08.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:07 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T22:33:08.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:07 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:08.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:07 smithi115 ceph-mon[23720]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:08.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:07 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T22:33:08.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:07 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:08.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:07 smithi115 ceph-mon[28216]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:08.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:07 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T22:33:08.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:07 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:09.014 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:08 smithi153 ceph-mon[24571]: Deploying daemon osd.6 on smithi153 2024-09-20T22:33:09.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:08 smithi115 ceph-mon[23720]: Deploying daemon osd.6 on smithi153 2024-09-20T22:33:09.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:08 smithi115 ceph-mon[28216]: Deploying daemon osd.6 on smithi153 2024-09-20T22:33:10.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:09 smithi153 ceph-mon[24571]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:10.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:09 smithi115 ceph-mon[23720]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:10.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:09 smithi115 ceph-mon[28216]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:11.911 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:11 smithi153 ceph-mon[24571]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:11 smithi115 ceph-mon[23720]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:12.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:11 smithi115 ceph-mon[28216]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:13 smithi115 ceph-mon[23720]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:13 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:13 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:13 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:13 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:14.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:13 smithi115 ceph-mon[28216]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:14.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:13 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:14.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:13 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:14.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:13 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:14.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:13 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:14.214 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:13 smithi153 ceph-mon[24571]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:14.214 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:13 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:14.214 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:13 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:14.214 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:13 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:14.214 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:13 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:15.966 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:15 smithi153 ceph-mon[24571]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:15.998 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 6 on host 'smithi153' 2024-09-20T22:33:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:15 smithi115 ceph-mon[23720]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:16.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:15 smithi115 ceph-mon[28216]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:16.667 DEBUG:teuthology.orchestra.run.smithi153:osd.6> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.6.service 2024-09-20T22:33:16.669 INFO:tasks.cephadm:Deploying osd.7 on smithi153 with /dev/vg_nvme/lv_1... 2024-09-20T22:33:16.670 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T22:33:16.902 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:16 smithi153 ceph-mon[24571]: from='osd.6 [v2:172.21.15.153:6816/868266364,v1:172.21.15.153:6817/868266364]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:33:16.902 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:16 smithi153 ceph-mon[24571]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:33:16.902 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:16 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:16 smithi115 ceph-mon[23720]: from='osd.6 [v2:172.21.15.153:6816/868266364,v1:172.21.15.153:6817/868266364]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:33:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:16 smithi115 ceph-mon[23720]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:33:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:16 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:16 smithi115 ceph-mon[28216]: from='osd.6 [v2:172.21.15.153:6816/868266364,v1:172.21.15.153:6817/868266364]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:33:17.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:16 smithi115 ceph-mon[28216]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:33:17.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:16 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:17.977 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:17 smithi153 ceph-mon[24571]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:17.978 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:17 smithi153 ceph-mon[24571]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T22:33:17.978 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:17 smithi153 ceph-mon[24571]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T22:33:17.978 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:17 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:17.978 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:17 smithi153 ceph-mon[24571]: from='osd.6 [v2:172.21.15.153:6816/868266364,v1:172.21.15.153:6817/868266364]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:33:17.978 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:17 smithi153 ceph-mon[24571]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:33:17.978 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:17 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:17.978 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:33:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[33719]: 2024-09-20T22:33:17.836+0000 7fd24fc70700 -1 osd.6 0 waiting for initial osdmap 2024-09-20T22:33:17.978 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:33:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[33719]: 2024-09-20T22:33:17.843+0000 7fd24c60b700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:33:18.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[23720]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:18.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[23720]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T22:33:18.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[23720]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T22:33:18.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:18.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[23720]: from='osd.6 [v2:172.21.15.153:6816/868266364,v1:172.21.15.153:6817/868266364]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:33:18.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[23720]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:33:18.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:18.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[28216]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:18.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[28216]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T22:33:18.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[28216]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T22:33:18.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:18.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[28216]: from='osd.6 [v2:172.21.15.153:6816/868266364,v1:172.21.15.153:6817/868266364]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:33:18.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[28216]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:33:18.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:17 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:18.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:18 smithi153 ceph-mon[24571]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-09-20T22:33:18.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:18 smithi153 ceph-mon[24571]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T22:33:18.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:18 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:18.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:18 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:18.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:18 smithi153 ceph-mon[24571]: osd.6 [v2:172.21.15.153:6816/868266364,v1:172.21.15.153:6817/868266364] boot 2024-09-20T22:33:18.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:18 smithi153 ceph-mon[24571]: osdmap e47: 7 total, 7 up, 7 in 2024-09-20T22:33:18.947 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:18 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:19.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[23720]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-09-20T22:33:19.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[23720]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T22:33:19.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:19.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:19.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[23720]: osd.6 [v2:172.21.15.153:6816/868266364,v1:172.21.15.153:6817/868266364] boot 2024-09-20T22:33:19.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[23720]: osdmap e47: 7 total, 7 up, 7 in 2024-09-20T22:33:19.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:19.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[28216]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-09-20T22:33:19.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[28216]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T22:33:19.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:19.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:19.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[28216]: osd.6 [v2:172.21.15.153:6816/868266364,v1:172.21.15.153:6817/868266364] boot 2024-09-20T22:33:19.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[28216]: osdmap e47: 7 total, 7 up, 7 in 2024-09-20T22:33:19.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:18 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:33:19.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:19 smithi153 ceph-mon[24571]: purged_snaps scrub starts 2024-09-20T22:33:19.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:19 smithi153 ceph-mon[24571]: purged_snaps scrub ok 2024-09-20T22:33:19.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:19 smithi153 ceph-mon[24571]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:19.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:19 smithi153 ceph-mon[24571]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T22:33:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:19 smithi115 ceph-mon[23720]: purged_snaps scrub starts 2024-09-20T22:33:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:19 smithi115 ceph-mon[23720]: purged_snaps scrub ok 2024-09-20T22:33:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:19 smithi115 ceph-mon[23720]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:19 smithi115 ceph-mon[23720]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T22:33:20.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:19 smithi115 ceph-mon[28216]: purged_snaps scrub starts 2024-09-20T22:33:20.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:19 smithi115 ceph-mon[28216]: purged_snaps scrub ok 2024-09-20T22:33:20.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:19 smithi115 ceph-mon[28216]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:33:20.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:19 smithi115 ceph-mon[28216]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T22:33:20.770 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-20T22:33:20.791 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi153:vg_nvme/lv_1 2024-09-20T22:33:21.901 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:21 smithi153 ceph-mon[24571]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:21.901 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:21 smithi153 ceph-mon[24571]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T22:33:21.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:21 smithi115 ceph-mon[23720]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:21.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:21 smithi115 ceph-mon[23720]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T22:33:21.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:21 smithi115 ceph-mon[28216]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:21.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:21 smithi115 ceph-mon[28216]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T22:33:22.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:22 smithi115 ceph-mon[23720]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T22:33:22.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:22 smithi115 ceph-mon[28216]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T22:33:23.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:22 smithi153 ceph-mon[24571]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T22:33:23.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:23.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T22:33:23.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: Detected new or changed devices on smithi153 2024-09-20T22:33:23.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:23.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:23.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:23.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: Adjusting osd_memory_target on smithi153 to 5696M 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: Detected new or changed devices on smithi153 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:23.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:23.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: Adjusting osd_memory_target on smithi153 to 5696M 2024-09-20T22:33:23.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:23.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:23.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:23.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:23.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:23.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:23 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:24.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T22:33:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: Detected new or changed devices on smithi153 2024-09-20T22:33:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: Adjusting osd_memory_target on smithi153 to 5696M 2024-09-20T22:33:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:23 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:24.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:24 smithi153 ceph-mon[24571]: from='client.34265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:24.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:24 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:33:24.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:24 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:33:24.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:24 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:24 smithi115 ceph-mon[23720]: from='client.34265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:24 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:33:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:24 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:33:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:24 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:25.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:24 smithi115 ceph-mon[28216]: from='client.34265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:25.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:24 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:33:25.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:24 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:33:25.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:24 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:26.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:25 smithi153 ceph-mon[24571]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-20T22:33:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:25 smithi115 ceph-mon[23720]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-20T22:33:26.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:25 smithi115 ceph-mon[28216]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-20T22:33:27.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:26 smithi153 ceph-mon[24571]: from='client.? 172.21.15.153:0/2241821943' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcd93d4d-70b2-4bb8-96f4-3d6e9d7005ba"}]: dispatch 2024-09-20T22:33:27.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:26 smithi153 ceph-mon[24571]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcd93d4d-70b2-4bb8-96f4-3d6e9d7005ba"}]: dispatch 2024-09-20T22:33:27.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:26 smithi153 ceph-mon[24571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcd93d4d-70b2-4bb8-96f4-3d6e9d7005ba"}]': finished 2024-09-20T22:33:27.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:26 smithi153 ceph-mon[24571]: osdmap e52: 8 total, 7 up, 8 in 2024-09-20T22:33:27.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:26 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:26 smithi115 ceph-mon[23720]: from='client.? 172.21.15.153:0/2241821943' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcd93d4d-70b2-4bb8-96f4-3d6e9d7005ba"}]: dispatch 2024-09-20T22:33:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:26 smithi115 ceph-mon[23720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcd93d4d-70b2-4bb8-96f4-3d6e9d7005ba"}]: dispatch 2024-09-20T22:33:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:26 smithi115 ceph-mon[23720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcd93d4d-70b2-4bb8-96f4-3d6e9d7005ba"}]': finished 2024-09-20T22:33:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:26 smithi115 ceph-mon[23720]: osdmap e52: 8 total, 7 up, 8 in 2024-09-20T22:33:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:26 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:27.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:26 smithi115 ceph-mon[28216]: from='client.? 172.21.15.153:0/2241821943' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcd93d4d-70b2-4bb8-96f4-3d6e9d7005ba"}]: dispatch 2024-09-20T22:33:27.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:26 smithi115 ceph-mon[28216]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcd93d4d-70b2-4bb8-96f4-3d6e9d7005ba"}]: dispatch 2024-09-20T22:33:27.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:26 smithi115 ceph-mon[28216]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcd93d4d-70b2-4bb8-96f4-3d6e9d7005ba"}]': finished 2024-09-20T22:33:27.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:26 smithi115 ceph-mon[28216]: osdmap e52: 8 total, 7 up, 8 in 2024-09-20T22:33:27.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:26 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:28.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:27 smithi153 ceph-mon[24571]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-20T22:33:28.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:27 smithi153 ceph-mon[24571]: from='client.? 172.21.15.153:0/464193215' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:33:28.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:27 smithi115 ceph-mon[23720]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-20T22:33:28.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:27 smithi115 ceph-mon[23720]: from='client.? 172.21.15.153:0/464193215' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:33:28.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:27 smithi115 ceph-mon[28216]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-20T22:33:28.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:27 smithi115 ceph-mon[28216]: from='client.? 172.21.15.153:0/464193215' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:33:30.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:29 smithi153 ceph-mon[24571]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-20T22:33:30.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:29 smithi115 ceph-mon[23720]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-20T22:33:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:29 smithi115 ceph-mon[28216]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-20T22:33:31.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:31 smithi153 ceph-mon[24571]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T22:33:32.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:31 smithi115 ceph-mon[23720]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T22:33:32.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:31 smithi115 ceph-mon[28216]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T22:33:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:33 smithi153 ceph-mon[24571]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-20T22:33:34.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T22:33:34.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:33 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:34.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:33 smithi153 ceph-mon[24571]: Deploying daemon osd.7 on smithi153 2024-09-20T22:33:34.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:33 smithi115 ceph-mon[23720]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-20T22:33:34.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T22:33:34.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:33 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:34.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:33 smithi115 ceph-mon[23720]: Deploying daemon osd.7 on smithi153 2024-09-20T22:33:34.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:33 smithi115 ceph-mon[28216]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-09-20T22:33:34.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T22:33:34.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:33 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:34.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:33 smithi115 ceph-mon[28216]: Deploying daemon osd.7 on smithi153 2024-09-20T22:33:35.956 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:35 smithi115 ceph-mon[23720]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:35.957 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:35 smithi115 ceph-mon[28216]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:36.000 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:35 smithi153 ceph-mon[24571]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:36.968 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:36 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:33:36.968 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:36 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:33:36.968 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:36 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:33:36.969 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:36 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:33:37.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:36 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:33:37.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:36 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:33:37.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:37 smithi153 ceph-mon[24571]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:38.139 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:37 smithi115 ceph-mon[23720]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:38.140 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:37 smithi115 ceph-mon[28216]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:39.224 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:38 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:39.224 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:38 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:39.224 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:38 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:39.224 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:38 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:39.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:38 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:39.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:38 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:39.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:38 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:39.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:38 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:39.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:38 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:39.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:38 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:39.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:38 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:39.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:38 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:40.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:39 smithi153 ceph-mon[24571]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:40.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:39 smithi115 ceph-mon[23720]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:40.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:39 smithi115 ceph-mon[28216]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:41.115 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 7 on host 'smithi153' 2024-09-20T22:33:41.925 DEBUG:teuthology.orchestra.run.smithi153:osd.7> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.7.service 2024-09-20T22:33:41.927 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-20T22:33:41.927 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T22:33:42.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:41 smithi153 ceph-mon[24571]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:42.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:41 smithi153 ceph-mon[24571]: from='osd.7 [v2:172.21.15.153:6824/2283577629,v1:172.21.15.153:6825/2283577629]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:33:42.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:41 smithi153 ceph-mon[24571]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:33:42.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:41 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:42.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:41 smithi115 ceph-mon[23720]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:42.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:41 smithi115 ceph-mon[23720]: from='osd.7 [v2:172.21.15.153:6824/2283577629,v1:172.21.15.153:6825/2283577629]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:33:42.230 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:41 smithi115 ceph-mon[23720]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:33:42.231 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:41 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:42.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:41 smithi115 ceph-mon[28216]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:42.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:41 smithi115 ceph-mon[28216]: from='osd.7 [v2:172.21.15.153:6824/2283577629,v1:172.21.15.153:6825/2283577629]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:33:42.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:41 smithi115 ceph-mon[28216]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:33:42.231 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:41 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:42 smithi153 ceph-mon[24571]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T22:33:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:42 smithi153 ceph-mon[24571]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T22:33:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:42 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:42 smithi153 ceph-mon[24571]: from='osd.7 [v2:172.21.15.153:6824/2283577629,v1:172.21.15.153:6825/2283577629]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:33:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:42 smithi153 ceph-mon[24571]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:33:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:42 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:43.030 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:33:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[36617]: 2024-09-20T22:33:42.944+0000 7fbdd3efb700 -1 osd.7 0 waiting for initial osdmap 2024-09-20T22:33:43.030 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:33:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[36617]: 2024-09-20T22:33:42.951+0000 7fbdcb88c700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:33:43.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:42 smithi115 ceph-mon[23720]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T22:33:43.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:42 smithi115 ceph-mon[23720]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T22:33:43.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:42 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:43.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:42 smithi115 ceph-mon[23720]: from='osd.7 [v2:172.21.15.153:6824/2283577629,v1:172.21.15.153:6825/2283577629]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:33:43.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:42 smithi115 ceph-mon[23720]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:33:43.234 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:42 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:43.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:42 smithi115 ceph-mon[28216]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T22:33:43.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:42 smithi115 ceph-mon[28216]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T22:33:43.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:42 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:43.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:42 smithi115 ceph-mon[28216]: from='osd.7 [v2:172.21.15.153:6824/2283577629,v1:172.21.15.153:6825/2283577629]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:33:43.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:42 smithi115 ceph-mon[28216]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:33:43.235 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:42 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:43.875 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:33:44.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[28216]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:44.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[28216]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-09-20T22:33:44.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[28216]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T22:33:44.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:44.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:44.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[28216]: osd.7 [v2:172.21.15.153:6824/2283577629,v1:172.21.15.153:6825/2283577629] boot 2024-09-20T22:33:44.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[28216]: osdmap e55: 8 total, 8 up, 8 in 2024-09-20T22:33:44.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:44.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1033226430' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T22:33:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[23720]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[23720]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-09-20T22:33:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[23720]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T22:33:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[23720]: osd.7 [v2:172.21.15.153:6824/2283577629,v1:172.21.15.153:6825/2283577629] boot 2024-09-20T22:33:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[23720]: osdmap e55: 8 total, 8 up, 8 in 2024-09-20T22:33:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:43 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1033226430' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T22:33:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:43 smithi153 ceph-mon[24571]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:33:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:43 smithi153 ceph-mon[24571]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-09-20T22:33:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:43 smithi153 ceph-mon[24571]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T22:33:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:43 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:43 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:43 smithi153 ceph-mon[24571]: osd.7 [v2:172.21.15.153:6824/2283577629,v1:172.21.15.153:6825/2283577629] boot 2024-09-20T22:33:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:43 smithi153 ceph-mon[24571]: osdmap e55: 8 total, 8 up, 8 in 2024-09-20T22:33:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:43 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:33:44.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:43 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1033226430' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T22:33:44.672 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1726871623,"num_in_osds":8,"osd_in_since":1726871606,"num_remapped_pgs":1} 2024-09-20T22:33:44.672 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T22:33:44.878 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:33:45.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:44 smithi153 ceph-mon[24571]: purged_snaps scrub starts 2024-09-20T22:33:45.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:44 smithi153 ceph-mon[24571]: purged_snaps scrub ok 2024-09-20T22:33:45.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:44 smithi153 ceph-mon[24571]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T22:33:45.210 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:44 smithi115 ceph-mon[23720]: purged_snaps scrub starts 2024-09-20T22:33:45.210 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:44 smithi115 ceph-mon[23720]: purged_snaps scrub ok 2024-09-20T22:33:45.210 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:44 smithi115 ceph-mon[23720]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T22:33:45.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:44 smithi115 ceph-mon[28216]: purged_snaps scrub starts 2024-09-20T22:33:45.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:44 smithi115 ceph-mon[28216]: purged_snaps scrub ok 2024-09-20T22:33:45.211 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:44 smithi115 ceph-mon[28216]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T22:33:46.225 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[23720]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:33:46.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:46.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:46.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:46.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:46.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:46.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:46.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:46.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:46.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:46.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:46.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:46.226 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[23720]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T22:33:46.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[28216]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:33:46.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:46.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:46.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:46.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:46.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:46.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:46.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:46.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:46.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:46.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:46.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:46.227 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:45 smithi115 ceph-mon[28216]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T22:33:46.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:45 smithi153 ceph-mon[24571]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:33:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:33:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:45 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:45 smithi153 ceph-mon[24571]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T22:33:46.643 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:33:46.643 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":57,"fsid":"9fe0f662-779f-11ef-baf5-efdc52797490","created":"2024-09-20T22:28:48.459373+0000","modified":"2024-09-20T22:33:45.647816+0000","last_up_change":"2024-09-20T22:33:43.640743+0000","last_in_change":"2024-09-20T22:33:26.106487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T22:31:46.724207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"5c2e890c-711f-4957-a306-2f3c211ab807","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":3107198721},{"type":"v1","addr":"172.21.15.115:6803","nonce":3107198721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":3107198721},{"type":"v1","addr":"172.21.15.115:6805","nonce":3107198721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":3107198721},{"type":"v1","addr":"172.21.15.115:6809","nonce":3107198721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":3107198721},{"type":"v1","addr":"172.21.15.115:6807","nonce":3107198721}]},"public_addr":"172.21.15.115:6803/3107198721","cluster_addr":"172.21.15.115:6805/3107198721","heartbeat_back_addr":"172.21.15.115:6809/3107198721","heartbeat_front_addr":"172.21.15.115:6807/3107198721","state":["exists","up"]},{"osd":1,"uuid":"b7592a9d-d11a-48c3-9cc9-2ddb6dcb2ac9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":650753341},{"type":"v1","addr":"172.21.15.115:6811","nonce":650753341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":650753341},{"type":"v1","addr":"172.21.15.115:6813","nonce":650753341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":650753341},{"type":"v1","addr":"172.21.15.115:6817","nonce":650753341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":650753341},{"type":"v1","addr":"172.21.15.115:6815","nonce":650753341}]},"public_addr":"172.21.15.115:6811/650753341","cluster_addr":"172.21.15.115:6813/650753341","heartbeat_back_addr":"172.21.15.115:6817/650753341","heartbeat_front_addr":"172.21.15.115:6815/650753341","state":["exists","up"]},{"osd":2,"uuid":"74152a6d-d11c-4405-b83f-248e8f225d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":123310734},{"type":"v1","addr":"172.21.15.115:6819","nonce":123310734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":123310734},{"type":"v1","addr":"172.21.15.115:6821","nonce":123310734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":123310734},{"type":"v1","addr":"172.21.15.115:6825","nonce":123310734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":123310734},{"type":"v1","addr":"172.21.15.115:6823","nonce":123310734}]},"public_addr":"172.21.15.115:6819/123310734","cluster_addr":"172.21.15.115:6821/123310734","heartbeat_back_addr":"172.21.15.115:6825/123310734","heartbeat_front_addr":"172.21.15.115:6823/123310734","state":["exists","up"]},{"osd":3,"uuid":"1ad34028-3e41-41c8-a7e5-4ff479f4287b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1168172919},{"type":"v1","addr":"172.21.15.115:6827","nonce":1168172919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1168172919},{"type":"v1","addr":"172.21.15.115:6829","nonce":1168172919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1168172919},{"type":"v1","addr":"172.21.15.115:6833","nonce":1168172919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1168172919},{"type":"v1","addr":"172.21.15.115:6831","nonce":1168172919}]},"public_addr":"172.21.15.115:6827/1168172919","cluster_addr":"172.21.15.115:6829/1168172919","heartbeat_back_addr":"172.21.15.115:6833/1168172919","heartbeat_front_addr":"172.21.15.115:6831/1168172919","state":["exists","up"]},{"osd":4,"uuid":"bffa26fb-977a-41d0-a298-8671fdb777e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":46581367},{"type":"v1","addr":"172.21.15.153:6801","nonce":46581367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":46581367},{"type":"v1","addr":"172.21.15.153:6803","nonce":46581367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":46581367},{"type":"v1","addr":"172.21.15.153:6807","nonce":46581367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":46581367},{"type":"v1","addr":"172.21.15.153:6805","nonce":46581367}]},"public_addr":"172.21.15.153:6801/46581367","cluster_addr":"172.21.15.153:6803/46581367","heartbeat_back_addr":"172.21.15.153:6807/46581367","heartbeat_front_addr":"172.21.15.153:6805/46581367","state":["exists","up"]},{"osd":5,"uuid":"cca50832-0a04-4e35-a109-dfe5da791cde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2463040153},{"type":"v1","addr":"172.21.15.153:6809","nonce":2463040153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2463040153},{"type":"v1","addr":"172.21.15.153:6811","nonce":2463040153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2463040153},{"type":"v1","addr":"172.21.15.153:6815","nonce":2463040153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2463040153},{"type":"v1","addr":"172.21.15.153:6813","nonce":2463040153}]},"public_addr":"172.21.15.153:6809/2463040153","cluster_addr":"172.21.15.153:6811/2463040153","heartbeat_back_addr":"172.21.15.153:6815/2463040153","heartbeat_front_addr":"172.21.15.153:6813/2463040153","state":["exists","up"]},{"osd":6,"uuid":"60e07fca-cfbd-45d5-a409-94f5e90afd26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":868266364},{"type":"v1","addr":"172.21.15.153:6817","nonce":868266364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":868266364},{"type":"v1","addr":"172.21.15.153:6819","nonce":868266364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":868266364},{"type":"v1","addr":"172.21.15.153:6823","nonce":868266364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":868266364},{"type":"v1","addr":"172.21.15.153:6821","nonce":868266364}]},"public_addr":"172.21.15.153:6817/868266364","cluster_addr":"172.21.15.153:6819/868266364","heartbeat_back_addr":"172.21.15.153:6823/868266364","heartbeat_front_addr":"172.21.15.153:6821/868266364","state":["exists","up"]},{"osd":7,"uuid":"fcd93d4d-70b2-4bb8-96f4-3d6e9d7005ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2283577629},{"type":"v1","addr":"172.21.15.153:6825","nonce":2283577629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2283577629},{"type":"v1","addr":"172.21.15.153:6827","nonce":2283577629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2283577629},{"type":"v1","addr":"172.21.15.153:6831","nonce":2283577629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2283577629},{"type":"v1","addr":"172.21.15.153:6829","nonce":2283577629}]},"public_addr":"172.21.15.153:6825/2283577629","cluster_addr":"172.21.15.153:6827/2283577629","heartbeat_back_addr":"172.21.15.153:6831/2283577629","heartbeat_front_addr":"172.21.15.153:6829/2283577629","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:30:54.389028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:31:19.239956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:31:43.117642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:32:07.021643+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:32:28.973912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:32:52.820858+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:33:16.894652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:33:42.056484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[0,6]}],"primary_temp":[],"blocklist":{"172.21.15.115:0/2869410039":"2024-09-21T22:29:36.365635+0000","172.21.15.115:6800/4280258613":"2024-09-21T22:29:36.365635+0000","172.21.15.115:0/3858670382":"2024-09-21T22:29:36.365635+0000","172.21.15.115:6801/4280258613":"2024-09-21T22:29:36.365635+0000","172.21.15.115:6800/2364546410":"2024-09-21T22:29:13.948916+0000","172.21.15.115:6801/2364546410":"2024-09-21T22:29:13.948916+0000","172.21.15.115:0/1642154865":"2024-09-21T22:29:13.948916+0000","172.21.15.115:0/1694194070":"2024-09-21T22:29:13.948916+0000","172.21.15.115:0/4062808784":"2024-09-21T22:29:36.365635+0000","172.21.15.115:0/267207316":"2024-09-21T22:29:13.948916+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T22:33:47.187 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:46 smithi115 ceph-mon[23720]: Detected new or changed devices on smithi153 2024-09-20T22:33:47.187 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:46 smithi115 ceph-mon[23720]: Adjusting osd_memory_target on smithi153 to 4272M 2024-09-20T22:33:47.187 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:46 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3522198358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:33:47.187 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:46 smithi115 ceph-mon[23720]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T22:33:47.187 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:46 smithi115 ceph-mon[28216]: Detected new or changed devices on smithi153 2024-09-20T22:33:47.187 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:46 smithi115 ceph-mon[28216]: Adjusting osd_memory_target on smithi153 to 4272M 2024-09-20T22:33:47.187 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:46 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3522198358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:33:47.188 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:46 smithi115 ceph-mon[28216]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T22:33:47.209 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T22:31:46.724207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-20T22:33:47.209 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-20T22:33:47.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:46 smithi153 ceph-mon[24571]: Detected new or changed devices on smithi153 2024-09-20T22:33:47.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:46 smithi153 ceph-mon[24571]: Adjusting osd_memory_target on smithi153 to 4272M 2024-09-20T22:33:47.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:46 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3522198358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:33:47.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:46 smithi153 ceph-mon[24571]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T22:33:47.412 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:33:48.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:47 smithi115 ceph-mon[23720]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:33:48.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:47 smithi115 ceph-mon[23720]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T22:33:48.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:47 smithi115 ceph-mon[28216]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:33:48.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:47 smithi115 ceph-mon[28216]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T22:33:48.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:47 smithi153 ceph-mon[24571]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:33:48.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:47 smithi153 ceph-mon[24571]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T22:33:49.088 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2024-09-20T22:33:49.780 INFO:tasks.cephadm:Adding prometheus.a on smithi153 2024-09-20T22:33:49.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch apply prometheus '1;smithi153=a' 2024-09-20T22:33:50.095 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:49 smithi153 ceph-mon[24571]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:33:50.095 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:49 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/285650927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T22:33:50.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:49 smithi115 ceph-mon[23720]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:33:50.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:49 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/285650927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T22:33:50.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:49 smithi115 ceph-mon[28216]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:33:50.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:49 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/285650927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T22:33:51.855 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled prometheus update... 2024-09-20T22:33:52.222 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:51 smithi153 ceph-mon[24571]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-20T22:33:52.222 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:52.222 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:52.222 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:52.223 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:52.223 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:52.223 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:51 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T22:33:52.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[23720]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-20T22:33:52.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:52.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:52.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:52.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:52.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:52.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T22:33:52.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[28216]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-20T22:33:52.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:52.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:33:52.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:33:52.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:33:52.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' 2024-09-20T22:33:52.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:51 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T22:33:52.538 DEBUG:teuthology.orchestra.run.smithi153:prometheus.a> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@prometheus.a.service 2024-09-20T22:33:52.541 INFO:tasks.cephadm:Adding node-exporter.a on smithi115 2024-09-20T22:33:52.541 INFO:tasks.cephadm:Adding node-exporter.b on smithi153 2024-09-20T22:33:52.541 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch apply node-exporter '2;smithi115=a;smithi153=b' 2024-09-20T22:33:53.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:52 smithi153 ceph-mon[24571]: from='client.34307 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:53.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:52 smithi153 ceph-mon[24571]: Saving service prometheus spec with placement smithi153=a;count:1 2024-09-20T22:33:53.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:52 smithi153 ceph-mon[24571]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T22:33:53.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:52 smithi153 ceph-mon[24571]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T22:33:53.202 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:52 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ignoring --setuser ceph since I am not root 2024-09-20T22:33:53.202 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:52 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ignoring --setgroup ceph since I am not root 2024-09-20T22:33:53.202 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:52 smithi153 ceph-mgr[25752]: -- 172.21.15.153:0/3021893383 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562370a2c340 con 0x562370a04c00 2024-09-20T22:33:53.202 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:53.039+0000 7f4e9ceb5000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:33:53.202 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:53.121+0000 7f4e9ceb5000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:33:53.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:52 smithi115 ceph-mon[28216]: from='client.34307 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:53.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:52 smithi115 ceph-mon[28216]: Saving service prometheus spec with placement smithi153=a;count:1 2024-09-20T22:33:53.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:52 smithi115 ceph-mon[28216]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T22:33:53.299 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:52 smithi115 ceph-mon[28216]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T22:33:53.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:52 smithi115 ceph-mon[23720]: from='client.34307 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:33:53.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:52 smithi115 ceph-mon[23720]: Saving service prometheus spec with placement smithi153=a;count:1 2024-09-20T22:33:53.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:52 smithi115 ceph-mon[23720]: from='mgr.14152 172.21.15.115:0/826909294' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T22:33:53.299 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:52 smithi115 ceph-mon[23720]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T22:33:53.299 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:52 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ignoring --setuser ceph since I am not root 2024-09-20T22:33:53.300 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:52 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ignoring --setgroup ceph since I am not root 2024-09-20T22:33:53.300 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:52 smithi115 ceph-mgr[23952]: -- 172.21.15.115:0/871450715 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e692af8340 con 0x55e692ad0c00 2024-09-20T22:33:53.300 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:53 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:53.044+0000 7fef986af000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:33:53.300 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:53 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:53.121+0000 7fef986af000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:33:53.300 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:53 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:53.296+0000 7fef986af000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:33:53.516 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:53.295+0000 7f4e9ceb5000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:33:53.516 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:53.382+0000 7f4e9ceb5000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:33:53.516 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:53.514+0000 7f4e9ceb5000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:33:53.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:53 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:53.384+0000 7fef986af000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:33:53.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:53 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:53.513+0000 7fef986af000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:33:53.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:53.726+0000 7f4e9ceb5000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:33:54.050 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:53 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:53.723+0000 7fef986af000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:33:54.106 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:54.061+0000 7f4e9ceb5000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:33:54.301 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:54 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:54.048+0000 7fef986af000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:33:54.301 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:54 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:54.135+0000 7fef986af000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:33:54.528 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:54.153+0000 7f4e9ceb5000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:33:54.529 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:54.321+0000 7f4e9ceb5000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:33:54.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:54 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:54.298+0000 7fef986af000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:33:55.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:55.371+0000 7fef986af000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:33:55.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:55.461+0000 7fef986af000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:33:55.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:55 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:55.372+0000 7f4e9ceb5000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:33:55.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:55 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:55.461+0000 7f4e9ceb5000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:33:56.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:56.360+0000 7fef986af000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:33:56.661 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:56 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:56.353+0000 7f4e9ceb5000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:33:56.979 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:56 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:56.659+0000 7f4e9ceb5000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:33:56.979 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:56 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:56.791+0000 7f4e9ceb5000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:33:56.979 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:56 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:56.879+0000 7f4e9ceb5000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:33:56.979 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:56 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:56.977+0000 7f4e9ceb5000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:33:56.990 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:56.666+0000 7fef986af000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:33:56.990 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:56.800+0000 7fef986af000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:33:56.990 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:56.889+0000 7fef986af000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:33:56.990 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:56.988+0000 7fef986af000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:33:57.278 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:57.064+0000 7f4e9ceb5000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:33:57.278 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:57.211+0000 7f4e9ceb5000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:33:57.321 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:57 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:57.076+0000 7fef986af000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:33:57.321 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:57 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:57.224+0000 7fef986af000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:33:57.321 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:57 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:57.318+0000 7fef986af000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:33:57.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:57 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:57.415+0000 7fef986af000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:33:57.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:57.305+0000 7f4e9ceb5000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:33:57.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:57.410+0000 7f4e9ceb5000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:33:58.028 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:57.694+0000 7f4e9ceb5000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:33:58.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:57.781+0000 7f4e9ceb5000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:33:58.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:57.872+0000 7f4e9ceb5000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:33:58.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:57 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:57.698+0000 7fef986af000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:33:58.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:57 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:57.788+0000 7fef986af000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:33:58.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:57 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:57.877+0000 7fef986af000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:33:59.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:58 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:58.936+0000 7f4e9ceb5000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:33:59.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:58 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: [20/Sep/2024:22:33:58] ENGINE Bus STARTING 2024-09-20T22:33:59.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: CherryPy Checker: 2024-09-20T22:33:59.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: The Application mounted at '' has an empty config. 2024-09-20T22:33:59.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: 2024-09-20T22:33:59.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: [20/Sep/2024:22:33:59] ENGINE Serving on http://:::9283 2024-09-20T22:33:59.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: [20/Sep/2024:22:33:59] ENGINE Bus STARTED 2024-09-20T22:33:59.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:58 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:58.944+0000 7fef986af000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:33:59.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[28216]: Standby manager daemon x restarted 2024-09-20T22:33:59.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[28216]: Standby manager daemon x started 2024-09-20T22:33:59.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/2075916850' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:33:59.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/2075916850' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:33:59.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/2075916850' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:33:59.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/2075916850' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:33:59.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[28216]: Active manager daemon y restarted 2024-09-20T22:33:59.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[28216]: Activating manager daemon y 2024-09-20T22:33:59.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[28216]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T22:33:59.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[23720]: Standby manager daemon x restarted 2024-09-20T22:33:59.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[23720]: Standby manager daemon x started 2024-09-20T22:33:59.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/2075916850' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:33:59.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/2075916850' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:33:59.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/2075916850' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:33:59.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/2075916850' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:33:59.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[23720]: Active manager daemon y restarted 2024-09-20T22:33:59.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[23720]: Activating manager daemon y 2024-09-20T22:33:59.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-mon[23720]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T22:33:59.906 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:33:59] ENGINE Bus STARTING 2024-09-20T22:33:59.906 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: CherryPy Checker: 2024-09-20T22:33:59.906 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: The Application mounted at '' has an empty config. 2024-09-20T22:33:59.906 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:33:59.906 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:33:59] ENGINE Serving on http://:::9283 2024-09-20T22:33:59.906 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:33:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:33:59] ENGINE Bus STARTED 2024-09-20T22:33:59.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-mon[24571]: Standby manager daemon x restarted 2024-09-20T22:33:59.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-mon[24571]: Standby manager daemon x started 2024-09-20T22:33:59.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/2075916850' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:33:59.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/2075916850' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:33:59.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/2075916850' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:33:59.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/2075916850' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:33:59.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-mon[24571]: Active manager daemon y restarted 2024-09-20T22:33:59.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-mon[24571]: Activating manager daemon y 2024-09-20T22:33:59.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:33:59 smithi153 ceph-mon[24571]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T22:34:00.674 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled node-exporter update... 2024-09-20T22:34:00.753 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:34:00] ENGINE Bus STARTING 2024-09-20T22:34:00.753 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:34:00] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:34:00.753 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:34:00] ENGINE Bus STARTED 2024-09-20T22:34:00.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: mgrmap e17: y(active, starting, since 0.710184s), standbys: x 2024-09-20T22:34:00.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:34:00.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:34:00.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:34:00.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:34:00.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:34:00.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:34:00.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:34:00.754 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:34:00.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:34:00.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:34:00.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:34:00.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:34:00.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:34:00.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:34:00.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:34:00.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:34:00.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: Manager daemon y is now available 2024-09-20T22:34:00.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:34:00.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:34:00.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:34:00.755 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:34:00.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:00.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:34:00.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:34:00.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:34:00.756 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:34:00.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: mgrmap e17: y(active, starting, since 0.710184s), standbys: x 2024-09-20T22:34:00.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:34:00.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:34:00.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:34:00.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:34:00.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:34:00.756 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: Manager daemon y is now available 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:34:00.757 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:34:00.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:34:00.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:00.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:34:00.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:34:00.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:34:00.758 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:00 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: mgrmap e17: y(active, starting, since 0.710184s), standbys: x 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:34:00.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:34:00.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:34:00.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:34:00.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:34:00.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: Manager daemon y is now available 2024-09-20T22:34:00.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:34:00.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:34:00.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:34:00.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:34:00.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:00.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:34:00.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:34:00.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:34:00.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:00 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:34:01.556 DEBUG:teuthology.orchestra.run.smithi115:node-exporter.a> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.a.service 2024-09-20T22:34:01.558 DEBUG:teuthology.orchestra.run.smithi153:node-exporter.b> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.b.service 2024-09-20T22:34:01.561 INFO:tasks.cephadm:Adding alertmanager.a on smithi115 2024-09-20T22:34:01.561 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch apply alertmanager '1;smithi115=a' 2024-09-20T22:34:01.704 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[23720]: [20/Sep/2024:22:34:00] ENGINE Bus STARTING 2024-09-20T22:34:01.704 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[23720]: mgrmap e18: y(active, since 1.71656s), standbys: x 2024-09-20T22:34:01.704 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[23720]: from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi115=a;smithi153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:01.704 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[23720]: Saving service node-exporter spec with placement smithi115=a;smithi153=b;count:2 2024-09-20T22:34:01.704 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[23720]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:01.704 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:01.704 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[23720]: [20/Sep/2024:22:34:00] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:34:01.704 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[23720]: [20/Sep/2024:22:34:00] ENGINE Bus STARTED 2024-09-20T22:34:01.704 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:01.704 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:01.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[28216]: [20/Sep/2024:22:34:00] ENGINE Bus STARTING 2024-09-20T22:34:01.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[28216]: mgrmap e18: y(active, since 1.71656s), standbys: x 2024-09-20T22:34:01.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[28216]: from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi115=a;smithi153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:01.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[28216]: Saving service node-exporter spec with placement smithi115=a;smithi153=b;count:2 2024-09-20T22:34:01.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[28216]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:01.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:01.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[28216]: [20/Sep/2024:22:34:00] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:34:01.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[28216]: [20/Sep/2024:22:34:00] ENGINE Bus STARTED 2024-09-20T22:34:01.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:01.705 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:01 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:01 smithi153 ceph-mon[24571]: [20/Sep/2024:22:34:00] ENGINE Bus STARTING 2024-09-20T22:34:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:01 smithi153 ceph-mon[24571]: mgrmap e18: y(active, since 1.71656s), standbys: x 2024-09-20T22:34:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:01 smithi153 ceph-mon[24571]: from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi115=a;smithi153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:01 smithi153 ceph-mon[24571]: Saving service node-exporter spec with placement smithi115=a;smithi153=b;count:2 2024-09-20T22:34:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:01 smithi153 ceph-mon[24571]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:01 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:01 smithi153 ceph-mon[24571]: [20/Sep/2024:22:34:00] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:34:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:01 smithi153 ceph-mon[24571]: [20/Sep/2024:22:34:00] ENGINE Bus STARTED 2024-09-20T22:34:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:01 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:01 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:02.826 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:02 smithi153 ceph-mon[24571]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:02.826 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:02 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:02.826 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:02 smithi153 ceph-mon[24571]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:34:02.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:02 smithi153 ceph-mon[24571]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-20T22:34:02.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:02 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:02.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:02 smithi153 ceph-mon[24571]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:34:02.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:02 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:02.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:02 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[23720]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[23720]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:34:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[23720]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-20T22:34:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:03.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[23720]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:34:03.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:03.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[28216]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[28216]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:34:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[28216]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-20T22:34:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[28216]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:34:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:02 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:03.826 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled alertmanager update... 2024-09-20T22:34:04.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:03 smithi153 ceph-mon[24571]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:34:04.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:03 smithi153 ceph-mon[24571]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:34:04.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:03 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:04.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:03 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:04.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:03 smithi153 ceph-mon[24571]: Deploying daemon prometheus.a on smithi153 2024-09-20T22:34:04.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:03 smithi153 ceph-mon[24571]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:04.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:03 smithi153 ceph-mon[24571]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi115=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:04.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:03 smithi153 ceph-mon[24571]: Saving service alertmanager spec with placement smithi115=a;count:1 2024-09-20T22:34:04.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:03 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:04.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[23720]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:34:04.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[23720]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:34:04.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:04.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:04.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[23720]: Deploying daemon prometheus.a on smithi153 2024-09-20T22:34:04.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[23720]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:04.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[23720]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi115=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:04.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[23720]: Saving service alertmanager spec with placement smithi115=a;count:1 2024-09-20T22:34:04.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:04.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[28216]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:34:04.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[28216]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:34:04.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:04.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:04.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[28216]: Deploying daemon prometheus.a on smithi153 2024-09-20T22:34:04.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[28216]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:04.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[28216]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi115=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:04.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[28216]: Saving service alertmanager spec with placement smithi115=a;count:1 2024-09-20T22:34:04.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:03 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:04.572 DEBUG:teuthology.orchestra.run.smithi115:alertmanager.a> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@alertmanager.a.service 2024-09-20T22:34:04.574 INFO:tasks.cephadm:Adding grafana.a on smithi153 2024-09-20T22:34:04.574 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph orch apply grafana '1;smithi153=a' 2024-09-20T22:34:06.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:06 smithi153 ceph-mon[24571]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:07.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:06 smithi115 ceph-mon[23720]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:07.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:06 smithi115 ceph-mon[28216]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:07.696 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled grafana update... 2024-09-20T22:34:08.683 DEBUG:teuthology.orchestra.run.smithi153:grafana.a> sudo journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@grafana.a.service 2024-09-20T22:34:08.686 INFO:tasks.cephadm:Setting up client nodes... 2024-09-20T22:34:08.686 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T22:34:08.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:08 smithi115 ceph-mon[28216]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:08.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:08 smithi115 ceph-mon[28216]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:08.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:08 smithi115 ceph-mon[28216]: Saving service grafana spec with placement smithi153=a;count:1 2024-09-20T22:34:08.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:08 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:08.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:08 smithi115 ceph-mon[23720]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:08.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:08 smithi115 ceph-mon[23720]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:08.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:08 smithi115 ceph-mon[23720]: Saving service grafana spec with placement smithi153=a;count:1 2024-09-20T22:34:08.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:08 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:09.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:08 smithi153 ceph-mon[24571]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:09.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:08 smithi153 ceph-mon[24571]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:34:09.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:08 smithi153 ceph-mon[24571]: Saving service grafana spec with placement smithi153=a;count:1 2024-09-20T22:34:09.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:08 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:10.684 INFO:teuthology.orchestra.run.smithi115.stdout:[client.0] 2024-09-20T22:34:10.685 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQBi+O1mR7GAKBAA/8eZHNuhrbsVFAWXAm19bA== 2024-09-20T22:34:11.014 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:10 smithi115 ceph-mon[23720]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:11.015 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:10 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/461366969' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:34:11.015 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:10 smithi115 ceph-mon[23720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:34:11.015 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:10 smithi115 ceph-mon[23720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T22:34:11.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:10 smithi115 ceph-mon[28216]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:11.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:10 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/461366969' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:34:11.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:10 smithi115 ceph-mon[28216]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:34:11.015 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:10 smithi115 ceph-mon[28216]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T22:34:11.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:10 smithi153 ceph-mon[24571]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:11.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:10 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/461366969' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:34:11.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:10 smithi153 ceph-mon[24571]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:34:11.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:10 smithi153 ceph-mon[24571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T22:34:11.271 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-09-20T22:34:11.271 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-20T22:34:11.271 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-20T22:34:11.309 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T22:34:13.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:12 smithi153 ceph-mon[24571]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:12 smithi115 ceph-mon[23720]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:13.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:12 smithi115 ceph-mon[28216]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:15.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:14 smithi153 ceph-mon[24571]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:15.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:14 smithi115 ceph-mon[23720]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:15.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:14 smithi115 ceph-mon[28216]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:16.459 INFO:teuthology.orchestra.run.smithi153.stdout:[client.1] 2024-09-20T22:34:16.459 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBo+O1mlOz8GhAA5mR/oh4zTRTIbv1l+FUBpA== 2024-09-20T22:34:16.589 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:16 smithi153 podman[39624]: 2024-09-20 22:34:16.570987705 +0000 UTC m=+0.141409156 container create e01d4cd3b2f5d69e793845aa8313723d96bafb86a611f8c7f63dbb749cc26d55 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:34:17.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:16 smithi153 ceph-mon[24571]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:17.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:16 smithi153 ceph-mon[24571]: from='client.? 172.21.15.153:0/2400117004' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:34:17.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:16 smithi153 ceph-mon[24571]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:34:17.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:16 smithi153 ceph-mon[24571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T22:34:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:16 smithi115 ceph-mon[23720]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:16 smithi115 ceph-mon[23720]: from='client.? 172.21.15.153:0/2400117004' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:34:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:16 smithi115 ceph-mon[23720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:34:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:16 smithi115 ceph-mon[23720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T22:34:17.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:16 smithi115 ceph-mon[28216]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:17.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:16 smithi115 ceph-mon[28216]: from='client.? 172.21.15.153:0/2400117004' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:34:17.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:16 smithi115 ceph-mon[28216]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:34:17.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:16 smithi115 ceph-mon[28216]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T22:34:17.400 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-20T22:34:17.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-20T22:34:17.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-20T22:34:17.438 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-20T22:34:17.439 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-20T22:34:17.439 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph mgr dump --format=json 2024-09-20T22:34:17.572 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 podman[39624]: 2024-09-20 22:34:17.54378701 +0000 UTC m=+1.114208467 container init e01d4cd3b2f5d69e793845aa8313723d96bafb86a611f8c7f63dbb749cc26d55 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:34:17.572 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 podman[39624]: 2024-09-20 22:34:17.546955025 +0000 UTC m=+1.117376477 container start e01d4cd3b2f5d69e793845aa8313723d96bafb86a611f8c7f63dbb749cc26d55 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:34:17.652 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:18.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.570Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-20T22:34:18.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.570Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-09-20T22:34:18.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.570Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-09-20T22:34:18.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.570Z caller=main.go:518 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi153 (none))" 2024-09-20T22:34:18.030 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.570Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T22:34:18.030 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.570Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T22:34:18.030 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.572Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T22:34:18.030 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.572Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-09-20T22:34:18.030 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.573Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-09-20T22:34:18.030 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.575Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T22:34:18.031 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.575Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.982µs 2024-09-20T22:34:18.031 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.575Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T22:34:18.031 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.575Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-20T22:34:18.031 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.575Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.859µs wal_replay_duration=176.363µs total_replay_duration=211.654µs 2024-09-20T22:34:18.031 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.576Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T22:34:18.031 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.576Z caller=main.go:947 level=info msg="TSDB started" 2024-09-20T22:34:18.031 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.576Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T22:34:18.032 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.591Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.590521ms db_storage=600ns remote_storage=1.23µs web_handler=231ns query_engine=768ns scrape=367.061µs scrape_sd=15.57µs notify=779ns notify_sd=1.271µs rules=15.040682ms 2024-09-20T22:34:18.032 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:34:17.592Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-09-20T22:34:18.032 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 bash[39624]: e01d4cd3b2f5d69e793845aa8313723d96bafb86a611f8c7f63dbb749cc26d55 2024-09-20T22:34:18.032 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:34:17 smithi153 systemd[1]: Started Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:34:19.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:18 smithi153 ceph-mon[24571]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:19.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:18 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:19.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:18 smithi153 ceph-mon[24571]: Deploying daemon node-exporter.a on smithi115 2024-09-20T22:34:19.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:18 smithi115 ceph-mon[23720]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:19.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:18 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:19.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:18 smithi115 ceph-mon[23720]: Deploying daemon node-exporter.a on smithi115 2024-09-20T22:34:19.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:18 smithi115 ceph-mon[28216]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:19.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:18 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:19.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:18 smithi115 ceph-mon[28216]: Deploying daemon node-exporter.a on smithi115 2024-09-20T22:34:19.521 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:34:20.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:19 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1117497247' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T22:34:20.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:19 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:19 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1117497247' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T22:34:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:19 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:20.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:19 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1117497247' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T22:34:20.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:19 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:20.252 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":19,"active_gid":14427,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":3218602138},{"type":"v1","addr":"172.21.15.115:6801","nonce":3218602138}]},"active_addr":"172.21.15.115:6801/3218602138","active_change":"2024-09-20T22:33:58.946355+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34313,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.115:8443/","prometheus":"http://172.21.15.115:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":2879365262}]},{"addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":4223966031}]},{"addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":1102177079}]},{"addrvec":[{"type":"v2","addr":"172.21.15.115:0","nonce":2826853796}]}]}} 2024-09-20T22:34:20.255 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-20T22:34:20.256 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-20T22:34:20.256 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T22:34:20.476 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:20.589 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:20 smithi115 bash[42853]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-09-20T22:34:20.897 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:20 smithi115 ceph-mon[23720]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:20.897 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:20 smithi115 ceph-mon[28216]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:20.898 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:20 smithi115 bash[42853]: Getting image source signatures 2024-09-20T22:34:20.898 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:20 smithi115 bash[42853]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-09-20T22:34:20.898 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:20 smithi115 bash[42853]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-09-20T22:34:20.898 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:20 smithi115 bash[42853]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-09-20T22:34:21.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:20 smithi153 ceph-mon[24571]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:22.752 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:22 smithi115 bash[42853]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-09-20T22:34:23.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:22 smithi153 ceph-mon[24571]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:23.061 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:34:23.061 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":60,"fsid":"9fe0f662-779f-11ef-baf5-efdc52797490","created":"2024-09-20T22:28:48.459373+0000","modified":"2024-09-20T22:33:58.945944+0000","last_up_change":"2024-09-20T22:33:43.640743+0000","last_in_change":"2024-09-20T22:33:26.106487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T22:31:46.724207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"5c2e890c-711f-4957-a306-2f3c211ab807","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":3107198721},{"type":"v1","addr":"172.21.15.115:6803","nonce":3107198721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":3107198721},{"type":"v1","addr":"172.21.15.115:6805","nonce":3107198721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":3107198721},{"type":"v1","addr":"172.21.15.115:6809","nonce":3107198721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":3107198721},{"type":"v1","addr":"172.21.15.115:6807","nonce":3107198721}]},"public_addr":"172.21.15.115:6803/3107198721","cluster_addr":"172.21.15.115:6805/3107198721","heartbeat_back_addr":"172.21.15.115:6809/3107198721","heartbeat_front_addr":"172.21.15.115:6807/3107198721","state":["exists","up"]},{"osd":1,"uuid":"b7592a9d-d11a-48c3-9cc9-2ddb6dcb2ac9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":650753341},{"type":"v1","addr":"172.21.15.115:6811","nonce":650753341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":650753341},{"type":"v1","addr":"172.21.15.115:6813","nonce":650753341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":650753341},{"type":"v1","addr":"172.21.15.115:6817","nonce":650753341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":650753341},{"type":"v1","addr":"172.21.15.115:6815","nonce":650753341}]},"public_addr":"172.21.15.115:6811/650753341","cluster_addr":"172.21.15.115:6813/650753341","heartbeat_back_addr":"172.21.15.115:6817/650753341","heartbeat_front_addr":"172.21.15.115:6815/650753341","state":["exists","up"]},{"osd":2,"uuid":"74152a6d-d11c-4405-b83f-248e8f225d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":123310734},{"type":"v1","addr":"172.21.15.115:6819","nonce":123310734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":123310734},{"type":"v1","addr":"172.21.15.115:6821","nonce":123310734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":123310734},{"type":"v1","addr":"172.21.15.115:6825","nonce":123310734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":123310734},{"type":"v1","addr":"172.21.15.115:6823","nonce":123310734}]},"public_addr":"172.21.15.115:6819/123310734","cluster_addr":"172.21.15.115:6821/123310734","heartbeat_back_addr":"172.21.15.115:6825/123310734","heartbeat_front_addr":"172.21.15.115:6823/123310734","state":["exists","up"]},{"osd":3,"uuid":"1ad34028-3e41-41c8-a7e5-4ff479f4287b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1168172919},{"type":"v1","addr":"172.21.15.115:6827","nonce":1168172919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1168172919},{"type":"v1","addr":"172.21.15.115:6829","nonce":1168172919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1168172919},{"type":"v1","addr":"172.21.15.115:6833","nonce":1168172919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1168172919},{"type":"v1","addr":"172.21.15.115:6831","nonce":1168172919}]},"public_addr":"172.21.15.115:6827/1168172919","cluster_addr":"172.21.15.115:6829/1168172919","heartbeat_back_addr":"172.21.15.115:6833/1168172919","heartbeat_front_addr":"172.21.15.115:6831/1168172919","state":["exists","up"]},{"osd":4,"uuid":"bffa26fb-977a-41d0-a298-8671fdb777e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":46581367},{"type":"v1","addr":"172.21.15.153:6801","nonce":46581367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":46581367},{"type":"v1","addr":"172.21.15.153:6803","nonce":46581367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":46581367},{"type":"v1","addr":"172.21.15.153:6807","nonce":46581367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":46581367},{"type":"v1","addr":"172.21.15.153:6805","nonce":46581367}]},"public_addr":"172.21.15.153:6801/46581367","cluster_addr":"172.21.15.153:6803/46581367","heartbeat_back_addr":"172.21.15.153:6807/46581367","heartbeat_front_addr":"172.21.15.153:6805/46581367","state":["exists","up"]},{"osd":5,"uuid":"cca50832-0a04-4e35-a109-dfe5da791cde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2463040153},{"type":"v1","addr":"172.21.15.153:6809","nonce":2463040153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2463040153},{"type":"v1","addr":"172.21.15.153:6811","nonce":2463040153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2463040153},{"type":"v1","addr":"172.21.15.153:6815","nonce":2463040153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2463040153},{"type":"v1","addr":"172.21.15.153:6813","nonce":2463040153}]},"public_addr":"172.21.15.153:6809/2463040153","cluster_addr":"172.21.15.153:6811/2463040153","heartbeat_back_addr":"172.21.15.153:6815/2463040153","heartbeat_front_addr":"172.21.15.153:6813/2463040153","state":["exists","up"]},{"osd":6,"uuid":"60e07fca-cfbd-45d5-a409-94f5e90afd26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":868266364},{"type":"v1","addr":"172.21.15.153:6817","nonce":868266364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":868266364},{"type":"v1","addr":"172.21.15.153:6819","nonce":868266364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":868266364},{"type":"v1","addr":"172.21.15.153:6823","nonce":868266364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":868266364},{"type":"v1","addr":"172.21.15.153:6821","nonce":868266364}]},"public_addr":"172.21.15.153:6817/868266364","cluster_addr":"172.21.15.153:6819/868266364","heartbeat_back_addr":"172.21.15.153:6823/868266364","heartbeat_front_addr":"172.21.15.153:6821/868266364","state":["exists","up"]},{"osd":7,"uuid":"fcd93d4d-70b2-4bb8-96f4-3d6e9d7005ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2283577629},{"type":"v1","addr":"172.21.15.153:6825","nonce":2283577629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2283577629},{"type":"v1","addr":"172.21.15.153:6827","nonce":2283577629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2283577629},{"type":"v1","addr":"172.21.15.153:6831","nonce":2283577629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2283577629},{"type":"v1","addr":"172.21.15.153:6829","nonce":2283577629}]},"public_addr":"172.21.15.153:6825/2283577629","cluster_addr":"172.21.15.153:6827/2283577629","heartbeat_back_addr":"172.21.15.153:6831/2283577629","heartbeat_front_addr":"172.21.15.153:6829/2283577629","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:30:54.389028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:31:19.239956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:31:43.117642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:32:07.021643+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:32:28.973912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:32:52.820858+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:33:16.894652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:33:42.056484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6801/3265629470":"2024-09-21T22:33:58.945900+0000","172.21.15.115:0/2244521742":"2024-09-21T22:33:58.945900+0000","172.21.15.115:0/2869410039":"2024-09-21T22:29:36.365635+0000","172.21.15.115:6800/4280258613":"2024-09-21T22:29:36.365635+0000","172.21.15.115:0/2928344409":"2024-09-21T22:33:58.945900+0000","172.21.15.115:0/3858670382":"2024-09-21T22:29:36.365635+0000","172.21.15.115:6800/3265629470":"2024-09-21T22:33:58.945900+0000","172.21.15.115:6801/4280258613":"2024-09-21T22:29:36.365635+0000","172.21.15.115:6800/2364546410":"2024-09-21T22:29:13.948916+0000","172.21.15.115:6801/2364546410":"2024-09-21T22:29:13.948916+0000","172.21.15.115:0/3595964724":"2024-09-21T22:33:58.945900+0000","172.21.15.115:0/1642154865":"2024-09-21T22:29:13.948916+0000","172.21.15.115:0/1694194070":"2024-09-21T22:29:13.948916+0000","172.21.15.115:0/804615791":"2024-09-21T22:33:58.945900+0000","172.21.15.115:0/4062808784":"2024-09-21T22:29:36.365635+0000","172.21.15.115:0/267207316":"2024-09-21T22:29:13.948916+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T22:34:23.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:22 smithi115 ceph-mon[23720]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:23.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:22 smithi115 ceph-mon[28216]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:23.062 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:22 smithi115 bash[42853]: Writing manifest to image destination 2024-09-20T22:34:23.404 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 podman[42853]: 2024-09-20 22:34:23.243610242 +0000 UTC m=+3.435950871 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-09-20T22:34:23.660 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 podman[42853]: 2024-09-20 22:34:23.575480607 +0000 UTC m=+3.767821237 container create 111c7bbee4de5f493d1ef9c9b72630e703c2717d63a1b590c540f0bcedb38409 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:34:24.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:23 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3762737676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:34:24.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3762737676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:34:24.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3762737676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:34:24.154 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 podman[42853]: 2024-09-20 22:34:23.761235734 +0000 UTC m=+3.953576373 container init 111c7bbee4de5f493d1ef9c9b72630e703c2717d63a1b590c540f0bcedb38409 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 podman[42853]: 2024-09-20 22:34:23.764706724 +0000 UTC m=+3.957047355 container start 111c7bbee4de5f493d1ef9c9b72630e703c2717d63a1b590c540f0bcedb38409 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.769Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.769Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.769Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.769Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=arp 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=bcache 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=bonding 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=btrfs 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=conntrack 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=cpu 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-09-20T22:34:24.155 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=diskstats 2024-09-20T22:34:24.156 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=dmi 2024-09-20T22:34:24.156 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=edac 2024-09-20T22:34:24.156 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=entropy 2024-09-20T22:34:24.156 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-09-20T22:34:24.156 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=filefd 2024-09-20T22:34:24.156 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=filesystem 2024-09-20T22:34:24.156 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=hwmon 2024-09-20T22:34:24.156 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=infiniband 2024-09-20T22:34:24.156 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=ipvs 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=loadavg 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=mdadm 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=meminfo 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=netclass 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=netdev 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=netstat 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=nfs 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=nfsd 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=nvme 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=os 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=pressure 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=rapl 2024-09-20T22:34:24.157 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=schedstat 2024-09-20T22:34:24.158 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=sockstat 2024-09-20T22:34:24.158 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=softnet 2024-09-20T22:34:24.158 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=stat 2024-09-20T22:34:24.158 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=tapestats 2024-09-20T22:34:24.158 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=textfile 2024-09-20T22:34:24.158 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-09-20T22:34:24.158 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=time 2024-09-20T22:34:24.158 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-09-20T22:34:24.158 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=uname 2024-09-20T22:34:24.158 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=vmstat 2024-09-20T22:34:24.158 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=xfs 2024-09-20T22:34:24.158 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:115 level=info collector=zfs 2024-09-20T22:34:24.159 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.770Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-09-20T22:34:24.159 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[43075]: ts=2024-09-20T22:34:23.771Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-09-20T22:34:24.159 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 bash[42853]: 111c7bbee4de5f493d1ef9c9b72630e703c2717d63a1b590c540f0bcedb38409 2024-09-20T22:34:24.159 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:34:23 smithi115 systemd[1]: Started Ceph node-exporter.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:34:24.365 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-20T22:34:24.365 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T22:34:24.572 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:24.925 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:24 smithi153 bash[39909]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-09-20T22:34:24.993 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:24 smithi115 ceph-mon[23720]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:24.993 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:24 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:24.993 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:24 smithi115 ceph-mon[23720]: Deploying daemon node-exporter.b on smithi153 2024-09-20T22:34:24.993 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:24 smithi115 ceph-mon[28216]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:24.993 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:24 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:24.993 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:24 smithi115 ceph-mon[28216]: Deploying daemon node-exporter.b on smithi153 2024-09-20T22:34:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:24 smithi153 ceph-mon[24571]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:24 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:24 smithi153 ceph-mon[24571]: Deploying daemon node-exporter.b on smithi153 2024-09-20T22:34:25.279 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:25 smithi153 bash[39909]: Getting image source signatures 2024-09-20T22:34:25.279 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:25 smithi153 bash[39909]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-09-20T22:34:25.279 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:25 smithi153 bash[39909]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-09-20T22:34:25.279 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:25 smithi153 bash[39909]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-09-20T22:34:26.469 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:34:26.469 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":60,"fsid":"9fe0f662-779f-11ef-baf5-efdc52797490","created":"2024-09-20T22:28:48.459373+0000","modified":"2024-09-20T22:33:58.945944+0000","last_up_change":"2024-09-20T22:33:43.640743+0000","last_in_change":"2024-09-20T22:33:26.106487+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T22:31:46.724207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"5c2e890c-711f-4957-a306-2f3c211ab807","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":3107198721},{"type":"v1","addr":"172.21.15.115:6803","nonce":3107198721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":3107198721},{"type":"v1","addr":"172.21.15.115:6805","nonce":3107198721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":3107198721},{"type":"v1","addr":"172.21.15.115:6809","nonce":3107198721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":3107198721},{"type":"v1","addr":"172.21.15.115:6807","nonce":3107198721}]},"public_addr":"172.21.15.115:6803/3107198721","cluster_addr":"172.21.15.115:6805/3107198721","heartbeat_back_addr":"172.21.15.115:6809/3107198721","heartbeat_front_addr":"172.21.15.115:6807/3107198721","state":["exists","up"]},{"osd":1,"uuid":"b7592a9d-d11a-48c3-9cc9-2ddb6dcb2ac9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":650753341},{"type":"v1","addr":"172.21.15.115:6811","nonce":650753341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":650753341},{"type":"v1","addr":"172.21.15.115:6813","nonce":650753341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":650753341},{"type":"v1","addr":"172.21.15.115:6817","nonce":650753341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":650753341},{"type":"v1","addr":"172.21.15.115:6815","nonce":650753341}]},"public_addr":"172.21.15.115:6811/650753341","cluster_addr":"172.21.15.115:6813/650753341","heartbeat_back_addr":"172.21.15.115:6817/650753341","heartbeat_front_addr":"172.21.15.115:6815/650753341","state":["exists","up"]},{"osd":2,"uuid":"74152a6d-d11c-4405-b83f-248e8f225d0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":123310734},{"type":"v1","addr":"172.21.15.115:6819","nonce":123310734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":123310734},{"type":"v1","addr":"172.21.15.115:6821","nonce":123310734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":123310734},{"type":"v1","addr":"172.21.15.115:6825","nonce":123310734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":123310734},{"type":"v1","addr":"172.21.15.115:6823","nonce":123310734}]},"public_addr":"172.21.15.115:6819/123310734","cluster_addr":"172.21.15.115:6821/123310734","heartbeat_back_addr":"172.21.15.115:6825/123310734","heartbeat_front_addr":"172.21.15.115:6823/123310734","state":["exists","up"]},{"osd":3,"uuid":"1ad34028-3e41-41c8-a7e5-4ff479f4287b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":1168172919},{"type":"v1","addr":"172.21.15.115:6827","nonce":1168172919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":1168172919},{"type":"v1","addr":"172.21.15.115:6829","nonce":1168172919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6832","nonce":1168172919},{"type":"v1","addr":"172.21.15.115:6833","nonce":1168172919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":1168172919},{"type":"v1","addr":"172.21.15.115:6831","nonce":1168172919}]},"public_addr":"172.21.15.115:6827/1168172919","cluster_addr":"172.21.15.115:6829/1168172919","heartbeat_back_addr":"172.21.15.115:6833/1168172919","heartbeat_front_addr":"172.21.15.115:6831/1168172919","state":["exists","up"]},{"osd":4,"uuid":"bffa26fb-977a-41d0-a298-8671fdb777e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":46581367},{"type":"v1","addr":"172.21.15.153:6801","nonce":46581367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":46581367},{"type":"v1","addr":"172.21.15.153:6803","nonce":46581367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":46581367},{"type":"v1","addr":"172.21.15.153:6807","nonce":46581367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":46581367},{"type":"v1","addr":"172.21.15.153:6805","nonce":46581367}]},"public_addr":"172.21.15.153:6801/46581367","cluster_addr":"172.21.15.153:6803/46581367","heartbeat_back_addr":"172.21.15.153:6807/46581367","heartbeat_front_addr":"172.21.15.153:6805/46581367","state":["exists","up"]},{"osd":5,"uuid":"cca50832-0a04-4e35-a109-dfe5da791cde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2463040153},{"type":"v1","addr":"172.21.15.153:6809","nonce":2463040153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2463040153},{"type":"v1","addr":"172.21.15.153:6811","nonce":2463040153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2463040153},{"type":"v1","addr":"172.21.15.153:6815","nonce":2463040153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2463040153},{"type":"v1","addr":"172.21.15.153:6813","nonce":2463040153}]},"public_addr":"172.21.15.153:6809/2463040153","cluster_addr":"172.21.15.153:6811/2463040153","heartbeat_back_addr":"172.21.15.153:6815/2463040153","heartbeat_front_addr":"172.21.15.153:6813/2463040153","state":["exists","up"]},{"osd":6,"uuid":"60e07fca-cfbd-45d5-a409-94f5e90afd26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":868266364},{"type":"v1","addr":"172.21.15.153:6817","nonce":868266364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":868266364},{"type":"v1","addr":"172.21.15.153:6819","nonce":868266364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":868266364},{"type":"v1","addr":"172.21.15.153:6823","nonce":868266364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":868266364},{"type":"v1","addr":"172.21.15.153:6821","nonce":868266364}]},"public_addr":"172.21.15.153:6817/868266364","cluster_addr":"172.21.15.153:6819/868266364","heartbeat_back_addr":"172.21.15.153:6823/868266364","heartbeat_front_addr":"172.21.15.153:6821/868266364","state":["exists","up"]},{"osd":7,"uuid":"fcd93d4d-70b2-4bb8-96f4-3d6e9d7005ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2283577629},{"type":"v1","addr":"172.21.15.153:6825","nonce":2283577629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2283577629},{"type":"v1","addr":"172.21.15.153:6827","nonce":2283577629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2283577629},{"type":"v1","addr":"172.21.15.153:6831","nonce":2283577629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2283577629},{"type":"v1","addr":"172.21.15.153:6829","nonce":2283577629}]},"public_addr":"172.21.15.153:6825/2283577629","cluster_addr":"172.21.15.153:6827/2283577629","heartbeat_back_addr":"172.21.15.153:6831/2283577629","heartbeat_front_addr":"172.21.15.153:6829/2283577629","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:30:54.389028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:31:19.239956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:31:43.117642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:32:07.021643+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:32:28.973912+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:32:52.820858+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:33:16.894652+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:33:42.056484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.115:6801/3265629470":"2024-09-21T22:33:58.945900+0000","172.21.15.115:0/2244521742":"2024-09-21T22:33:58.945900+0000","172.21.15.115:0/2869410039":"2024-09-21T22:29:36.365635+0000","172.21.15.115:6800/4280258613":"2024-09-21T22:29:36.365635+0000","172.21.15.115:0/2928344409":"2024-09-21T22:33:58.945900+0000","172.21.15.115:0/3858670382":"2024-09-21T22:29:36.365635+0000","172.21.15.115:6800/3265629470":"2024-09-21T22:33:58.945900+0000","172.21.15.115:6801/4280258613":"2024-09-21T22:29:36.365635+0000","172.21.15.115:6800/2364546410":"2024-09-21T22:29:13.948916+0000","172.21.15.115:6801/2364546410":"2024-09-21T22:29:13.948916+0000","172.21.15.115:0/3595964724":"2024-09-21T22:33:58.945900+0000","172.21.15.115:0/1642154865":"2024-09-21T22:29:13.948916+0000","172.21.15.115:0/1694194070":"2024-09-21T22:29:13.948916+0000","172.21.15.115:0/804615791":"2024-09-21T22:33:58.945900+0000","172.21.15.115:0/4062808784":"2024-09-21T22:29:36.365635+0000","172.21.15.115:0/267207316":"2024-09-21T22:29:13.948916+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T22:34:26.779 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:26 smithi153 bash[39909]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-09-20T22:34:26.779 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:26 smithi153 bash[39909]: Writing manifest to image destination 2024-09-20T22:34:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:26 smithi115 ceph-mon[23720]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:26 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1412385800' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:34:27.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:26 smithi115 ceph-mon[28216]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:27.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:26 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1412385800' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:34:27.219 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-20T22:34:27.241 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-20T22:34:27.241 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-20T22:34:27.242 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-20T22:34:27.242 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-20T22:34:27.242 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-20T22:34:27.243 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-20T22:34:27.243 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-20T22:34:27.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:26 smithi153 ceph-mon[24571]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:27.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:26 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1412385800' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:34:27.261 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 podman[39909]: 2024-09-20 22:34:26.908753802 +0000 UTC m=+2.235430055 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-09-20T22:34:27.261 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 podman[39909]: 2024-09-20 22:34:27.023932578 +0000 UTC m=+2.350608829 container create 5717e9268a3a6c2ac8a45b453a01b490189868675607c98ce070fb95c44b323c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:34:27.530 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 podman[39909]: 2024-09-20 22:34:27.260044305 +0000 UTC m=+2.586720566 container init 5717e9268a3a6c2ac8a45b453a01b490189868675607c98ce070fb95c44b323c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:34:27.530 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 podman[39909]: 2024-09-20 22:34:27.262913637 +0000 UTC m=+2.589589888 container start 5717e9268a3a6c2ac8a45b453a01b490189868675607c98ce070fb95c44b323c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:34:27.530 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.266Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-09-20T22:34:27.530 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.266Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-09-20T22:34:27.530 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-09-20T22:34:27.530 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T22:34:27.531 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-09-20T22:34:27.531 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=arp 2024-09-20T22:34:27.531 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=bcache 2024-09-20T22:34:27.531 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=bonding 2024-09-20T22:34:27.531 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=btrfs 2024-09-20T22:34:27.531 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=conntrack 2024-09-20T22:34:27.531 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=cpu 2024-09-20T22:34:27.532 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-09-20T22:34:27.532 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=diskstats 2024-09-20T22:34:27.532 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=dmi 2024-09-20T22:34:27.532 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=edac 2024-09-20T22:34:27.532 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=entropy 2024-09-20T22:34:27.532 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-09-20T22:34:27.532 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=filefd 2024-09-20T22:34:27.532 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=filesystem 2024-09-20T22:34:27.532 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=hwmon 2024-09-20T22:34:27.532 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=infiniband 2024-09-20T22:34:27.532 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=ipvs 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=loadavg 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=mdadm 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=meminfo 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=netclass 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=netdev 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=netstat 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=nfs 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=nfsd 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=nvme 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=os 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=pressure 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=rapl 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=schedstat 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=sockstat 2024-09-20T22:34:27.533 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=softnet 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=stat 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=tapestats 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=textfile 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=time 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=uname 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=vmstat 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=xfs 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:115 level=info collector=zfs 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.267Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[39986]: ts=2024-09-20T22:34:27.268Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 bash[39909]: 5717e9268a3a6c2ac8a45b453a01b490189868675607c98ce070fb95c44b323c 2024-09-20T22:34:27.534 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:34:27 smithi153 systemd[1]: Started Ceph node-exporter.b for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:34:27.682 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:27.682 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:27.685 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:27.688 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:27.689 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:27.694 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:27.696 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:27.699 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:28.524 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:28 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:28.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:28 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:34:28.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:28 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:34:28.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:28 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:34:28.525 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:28 smithi115 ceph-mon[23720]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:28.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:28 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:28.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:28 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:34:28.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:28 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:34:28.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:28 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:34:28.525 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:28 smithi115 ceph-mon[28216]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:28.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:28 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:28.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:28 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:34:28.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:28 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:34:28.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:28 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:34:28.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:28 smithi153 ceph-mon[24571]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:29.720 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:34:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:34:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:34:30.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:30 smithi115 ceph-mon[28216]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:30.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:30 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:30.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:30 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:30.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:30 smithi115 ceph-mon[23720]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:30.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:30 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:30.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:30 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:31.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:30 smithi153 ceph-mon[24571]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:31.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:30 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:31.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:30 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:34:31.568 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:34:31 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:34:31] "GET /metrics HTTP/1.1" 200 191517 "" "Prometheus/2.33.4" 2024-09-20T22:34:32.874 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:32 smithi115 ceph-mon[23720]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:32.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:32 smithi115 ceph-mon[28216]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:33.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:32 smithi153 ceph-mon[24571]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:34 smithi115 ceph-mon[23720]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:34.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:34 smithi115 ceph-mon[28216]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:34 smithi153 ceph-mon[24571]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:36.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:36 smithi115 ceph-mon[23720]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:36.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:36 smithi115 ceph-mon[28216]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:37.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:36 smithi153 ceph-mon[24571]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:37.961 INFO:teuthology.orchestra.run.smithi115.stdout:201863462929 2024-09-20T22:34:37.962 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-20T22:34:38.765 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:38 smithi115 ceph-mon[23720]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:38.765 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:38 smithi115 ceph-mon[28216]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:39.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:38 smithi153 ceph-mon[24571]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:39.351 INFO:teuthology.orchestra.run.smithi115.stdout:236223201292 2024-09-20T22:34:39.351 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-20T22:34:39.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:34:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:34:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:34:39.986 INFO:teuthology.orchestra.run.smithi115.stdout:133143986204 2024-09-20T22:34:39.987 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-20T22:34:40.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:40 smithi115 ceph-mon[23720]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:40.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:40 smithi115 ceph-mon[28216]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:41.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:40 smithi153 ceph-mon[24571]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:41.243 INFO:teuthology.orchestra.run.smithi115.stdout:55834574889 2024-09-20T22:34:41.244 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-20T22:34:41.330 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:34:41 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:34:41] "GET /metrics HTTP/1.1" 200 191517 "" "Prometheus/2.33.4" 2024-09-20T22:34:42.555 INFO:teuthology.orchestra.run.smithi115.stdout:167503724567 2024-09-20T22:34:42.555 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-20T22:34:42.779 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:42 smithi115 ceph-mon[23720]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:42.780 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:42 smithi115 ceph-mon[28216]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:43.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:42 smithi153 ceph-mon[24571]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:43.482 INFO:teuthology.orchestra.run.smithi115.stdout:34359738416 2024-09-20T22:34:43.482 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-20T22:34:43.708 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:43.710 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:43.712 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:44.302 INFO:teuthology.orchestra.run.smithi115.stdout:111669149728 2024-09-20T22:34:44.303 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-20T22:34:44.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:44 smithi115 ceph-mon[23720]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:44.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:44 smithi115 ceph-mon[28216]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:45.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:44 smithi153 ceph-mon[24571]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:45.425 INFO:teuthology.orchestra.run.smithi115.stdout:77309411366 2024-09-20T22:34:45.425 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-20T22:34:45.893 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:45.896 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:45.897 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:45.897 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:47.005 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:46 smithi115 ceph-mon[23720]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:47.005 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:46 smithi115 ceph-mon[28216]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:47.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:46 smithi153 ceph-mon[24571]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:47.585 INFO:teuthology.orchestra.run.smithi115.stdout:201863462930 2024-09-20T22:34:47.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:47 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1907030684' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T22:34:47.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:47 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1907030684' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T22:34:48.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:47 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1907030684' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T22:34:49.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:48 smithi153 ceph-mon[24571]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:49.107 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:48 smithi115 ceph-mon[23720]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:49.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:48 smithi115 ceph-mon[28216]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:49.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:34:49 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:34:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:34:50.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:50 smithi115 ceph-mon[23720]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:50.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:50 smithi115 ceph-mon[28216]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:51.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:50 smithi153 ceph-mon[24571]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:51.228 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462929 got 201863462930 for osd.6 2024-09-20T22:34:51.228 DEBUG:teuthology.parallel:result is None 2024-09-20T22:34:51.489 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:34:51 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:34:51] "GET /metrics HTTP/1.1" 200 191498 "" "Prometheus/2.33.4" 2024-09-20T22:34:52.619 INFO:teuthology.orchestra.run.smithi115.stdout:111669149731 2024-09-20T22:34:53.018 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:52 smithi115 ceph-mon[23720]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:53.018 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:52 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/500716856' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T22:34:53.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:52 smithi115 ceph-mon[28216]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:53.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:52 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/500716856' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T22:34:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:52 smithi153 ceph-mon[24571]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:52 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/500716856' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T22:34:54.875 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:54 smithi115 ceph-mon[23720]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:54.876 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:54 smithi115 ceph-mon[28216]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:55.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:54 smithi153 ceph-mon[24571]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:55.059 INFO:teuthology.orchestra.run.smithi115.stdout:34359738418 2024-09-20T22:34:55.674 INFO:teuthology.orchestra.run.smithi115.stdout:236223201295 2024-09-20T22:34:55.894 INFO:teuthology.orchestra.run.smithi115.stdout:55834574893 2024-09-20T22:34:55.938 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:34:55.940 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:55 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3760209051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T22:34:55.940 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:55 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3760209051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T22:34:56.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:55 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3760209051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T22:34:56.552 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149728 got 111669149731 for osd.3 2024-09-20T22:34:56.552 DEBUG:teuthology.parallel:result is None 2024-09-20T22:34:56.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:56 smithi115 ceph-mon[23720]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:56.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:56 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/4220588488' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T22:34:56.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:56 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3816860473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T22:34:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:56 smithi115 ceph-mon[28216]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:56 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/4220588488' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T22:34:56.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:56 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3816860473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T22:34:57.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:56 smithi153 ceph-mon[24571]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:56 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/4220588488' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T22:34:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:56 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3816860473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T22:34:57.481 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738418 for osd.0 2024-09-20T22:34:57.482 DEBUG:teuthology.parallel:result is None 2024-09-20T22:34:57.800 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574893 for osd.1 2024-09-20T22:34:57.800 DEBUG:teuthology.parallel:result is None 2024-09-20T22:34:58.163 INFO:teuthology.orchestra.run.smithi115.stdout:133143986208 2024-09-20T22:34:58.505 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201292 got 236223201295 for osd.7 2024-09-20T22:34:58.505 DEBUG:teuthology.parallel:result is None 2024-09-20T22:34:58.821 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:58 smithi115 ceph-mon[23720]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:58.821 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:58 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2444432963' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T22:34:58.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:58 smithi115 ceph-mon[28216]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:58.821 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:58 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2444432963' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T22:34:59.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:58 smithi153 ceph-mon[24571]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:34:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:58 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2444432963' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T22:34:59.319 INFO:teuthology.orchestra.run.smithi115.stdout:167503724570 2024-09-20T22:34:59.600 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986204 got 133143986208 for osd.4 2024-09-20T22:34:59.600 DEBUG:teuthology.parallel:result is None 2024-09-20T22:34:59.731 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:34:59 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:34:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:34:59.885 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:59 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2075486092' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T22:34:59.885 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:59 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:34:59.885 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:34:59 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:34:59.885 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:59 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2075486092' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T22:34:59.885 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:59 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:34:59.886 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:34:59 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:35:00.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:59 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2075486092' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T22:35:00.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:59 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:35:00.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:34:59 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:35:00.677 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724567 got 167503724570 for osd.5 2024-09-20T22:35:00.677 DEBUG:teuthology.parallel:result is None 2024-09-20T22:35:00.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:00 smithi115 ceph-mon[28216]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:00.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:35:00.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:00 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:35:00.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:00 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:00.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:00 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:00.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:00 smithi115 ceph-mon[28216]: Deploying daemon alertmanager.a on smithi115 2024-09-20T22:35:00.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:00 smithi115 ceph-mon[23720]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:00.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:35:00.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:00 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:35:00.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:00 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:00.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:00 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:00.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:00 smithi115 ceph-mon[23720]: Deploying daemon alertmanager.a on smithi115 2024-09-20T22:35:01.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:00 smithi153 ceph-mon[24571]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:01.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:35:01.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:00 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:35:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:00 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:00 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:00 smithi153 ceph-mon[24571]: Deploying daemon alertmanager.a on smithi115 2024-09-20T22:35:01.358 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:35:01 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:35:01] "GET /metrics HTTP/1.1" 200 191503 "" "Prometheus/2.33.4" 2024-09-20T22:35:02.400 INFO:teuthology.orchestra.run.smithi115.stdout:77309411370 2024-09-20T22:35:03.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:02 smithi153 ceph-mon[24571]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:03.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:02 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3335458934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T22:35:03.107 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:02 smithi115 ceph-mon[23720]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:03.107 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:02 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3335458934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T22:35:03.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:02 smithi115 ceph-mon[28216]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:03.107 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:02 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3335458934' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T22:35:03.133 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411366 got 77309411370 for osd.2 2024-09-20T22:35:03.133 DEBUG:teuthology.parallel:result is None 2024-09-20T22:35:03.133 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-20T22:35:03.133 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:35:03.344 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:35:05.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:04 smithi153 ceph-mon[24571]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:05.121 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:04 smithi115 ceph-mon[23720]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:05.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:04 smithi115 ceph-mon[28216]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:06.995 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:06 smithi115 ceph-mon[23720]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:06.995 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:06 smithi115 ceph-mon[28216]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:07.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:06 smithi153 ceph-mon[24571]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:08.588 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:35:08.590 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2024-09-20T22:35:08.827 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:08 smithi115 ceph-mon[23720]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:08.827 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:08 smithi115 ceph-mon[28216]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:09.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:08 smithi153 ceph-mon[24571]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:09.404 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:09 smithi115 systemd[1]: Starting Ceph alertmanager.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:35:09.735 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:35:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:35:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:35:09.771 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-09-20T22:35:07.673930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48900,"kb_used_data":6148,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912316,"statfs":{"total":767960285184,"available":767910211584,"internally_reserved":0,"allocated":6295552,"data_stored":3510393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002478"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-20T22:33:59.749334+0000","last_change":"2024-09-20T22:33:47.659386+0000","last_active":"2024-09-20T22:33:59.749334+0000","last_peered":"2024-09-20T22:33:59.749334+0000","last_clean":"2024-09-20T22:33:59.749334+0000","last_became_active":"2024-09-20T22:33:47.659042+0000","last_became_peered":"2024-09-20T22:33:47.659042+0000","last_unstale":"2024-09-20T22:33:59.749334+0000","last_undegraded":"2024-09-20T22:33:59.749334+0000","last_fullsized":"2024-09-20T22:33:59.749334+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:31:47.021654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:31:47.021654+0000","last_clean_scrub_stamp":"2024-09-20T22:31:47.021654+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:14:59.291958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6092,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739060,"statfs":{"total":95995035648,"available":95988797440,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.48199999999999998}]},{"osd":1,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.501},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.42799999999999999}]},{"osd":2,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55000000000000004}]},{"osd":3,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.45800000000000002}]},{"osd":4,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.37},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.40699999999999997}]},{"osd":5,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.54100000000000004}]},{"osd":6,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.58599999999999997}]}]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.627}]},{"osd":1,"last update":"Fri Sep 20 22:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.56499999999999995}]},{"osd":2,"last update":"Fri Sep 20 22:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.66000000000000003}]},{"osd":3,"last update":"Fri Sep 20 22:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.68700000000000006}]},{"osd":4,"last update":"Fri Sep 20 22:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.64300000000000002}]},{"osd":5,"last update":"Fri Sep 20 22:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.67900000000000005}]},{"osd":7,"last update":"Fri Sep 20 22:34:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.746},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.69999999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:30 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.33800000000000002,"15min":0.317},"min":{"1min":0.22500000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.35199999999999998,"15min":0.32600000000000001},"min":{"1min":0.27500000000000002,"5min":0.192,"15min":0.192},"max":{"1min":0.52200000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.28799999999999998}]},{"osd":2,"last update":"Fri Sep 20 22:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.38600000000000001,"15min":0.37},"min":{"1min":0.29799999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.58699999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.376},"min":{"1min":0.25700000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.33700000000000002}]},{"osd":3,"last update":"Fri Sep 20 22:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42499999999999999,"15min":0.42899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44800000000000001,"15min":0.45100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.31}]},{"osd":4,"last update":"Fri Sep 20 22:34:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0029999999999999,"5min":1.718,"15min":1.718},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.434,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88800000000000001,"5min":1.671,"15min":1.671},"last":0.55300000000000005}]},{"osd":5,"last update":"Fri Sep 20 22:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.58599999999999997,"15min":0.57999999999999996},"min":{"1min":0.42999999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.82999999999999996,"5min":0.87,"15min":0.87},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.61099999999999999,"15min":0.60299999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.67400000000000004}]},{"osd":6,"last update":"Fri Sep 20 22:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.60999999999999999}]},{"osd":7,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.65600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 20 22:34:30 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.33100000000000002,"15min":0.313},"min":{"1min":0.23799999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.35699999999999998,"15min":0.33900000000000002},"min":{"1min":0.221,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.32100000000000001}]},{"osd":2,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.38200000000000001,"15min":0.373},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.45100000000000001,"5min":0.627,"15min":0.627},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.35899999999999999,"15min":0.35399999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.497,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.36199999999999999}]},{"osd":3,"last update":"Fri Sep 20 22:34:14 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.42299999999999999}]},{"osd":4,"last update":"Fri Sep 20 22:34:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.56799999999999995,"15min":0.57699999999999996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66600000000000004,"5min":1.137,"15min":1.137},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71599999999999997,"5min":1.079,"15min":1.079},"last":0.51500000000000001}]},{"osd":5,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.41899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54300000000000004,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.67600000000000005}]},{"osd":6,"last update":"Fri Sep 20 22:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.70199999999999996}]},{"osd":7,"last update":"Fri Sep 20 22:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.621}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":239784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.37,"15min":0.36199999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.375,"15min":0.378},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.52800000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.39900000000000002}]},{"osd":1,"last update":"Fri Sep 20 22:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39200000000000002,"15min":0.38800000000000001},"min":{"1min":0.28000000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.39100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.55900000000000005,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.38600000000000001}]},{"osd":3,"last update":"Fri Sep 20 22:34:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.439,"15min":0.44},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.374},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.34000000000000002}]},{"osd":4,"last update":"Fri Sep 20 22:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.67700000000000005,"15min":0.625},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":8.6180000000000003,"5min":8.6180000000000003,"15min":8.6180000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81799999999999995,"5min":1.194,"15min":1.194},"last":0.51500000000000001}]},{"osd":5,"last update":"Fri Sep 20 22:34:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.41799999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80900000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40899999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.60899999999999999}]},{"osd":6,"last update":"Fri Sep 20 22:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.66700000000000004}]},{"osd":7,"last update":"Fri Sep 20 22:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.64100000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.35799999999999998,"15min":0.35999999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.245},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.39600000000000002,"15min":0.39500000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.313}]},{"osd":1,"last update":"Fri Sep 20 22:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.433,"15min":0.441},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.33800000000000002}]},{"osd":2,"last update":"Fri Sep 20 22:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42299999999999999,"15min":0.42199999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.441,"15min":0.44500000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.28299999999999997}]},{"osd":4,"last update":"Fri Sep 20 22:34:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.55400000000000005,"15min":0.56100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.63100000000000001,"15min":0.64500000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0249999999999999,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":0.627}]},{"osd":5,"last update":"Fri Sep 20 22:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65400000000000003,"15min":0.66000000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.1080000000000001,"5min":1.841,"15min":1.841},"last":0.52900000000000003}]},{"osd":6,"last update":"Fri Sep 20 22:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.67000000000000004}]},{"osd":7,"last update":"Fri Sep 20 22:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.79900000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52700000000000002,"15min":0.52600000000000002},"min":{"1min":0.377,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67600000000000005,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.437},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.55400000000000005,"15min":0.56599999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65700000000000003,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.45900000000000002}]},{"osd":1,"last update":"Fri Sep 20 22:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53700000000000003,"15min":0.54300000000000004},"min":{"1min":0.36599999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.67900000000000005,"5min":0.877,"15min":0.877},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54500000000000004,"15min":0.55100000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68200000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.52100000000000002}]},{"osd":2,"last update":"Fri Sep 20 22:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.58799999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.38300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.747}]},{"osd":3,"last update":"Fri Sep 20 22:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.69699999999999995,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.42999999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.70699999999999996,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.64800000000000002}]},{"osd":5,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.42299999999999999,"15min":0.41899999999999998},"min":{"1min":0.33900000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56899999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.83099999999999996}]},{"osd":6,"last update":"Fri Sep 20 22:34:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.67500000000000004}]},{"osd":7,"last update":"Fri Sep 20 22:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.78500000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5636,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739516,"statfs":{"total":95995035648,"available":95989264384,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.378,"5min":0.372,"15min":0.372},"max":{"1min":0.66600000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.5},"min":{"1min":0.33300000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.52900000000000003}]},{"osd":1,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.68999999999999995,"5min":0.871,"15min":0.871},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.56100000000000005,"15min":0.56699999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.67800000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.49299999999999999}]},{"osd":2,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.54500000000000004,"15min":0.52000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.8470000000000004,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.39400000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.66700000000000004,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.45500000000000002}]},{"osd":3,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.40500000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73599999999999999,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.57399999999999995}]},{"osd":4,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.38900000000000001,"15min":0.377},"min":{"1min":0.27100000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.436},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.41599999999999998,"15min":0.40600000000000003},"min":{"1min":0.28100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.67200000000000004,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56499999999999995}]},{"osd":6,"last update":"Fri Sep 20 22:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.501}]},{"osd":7,"last update":"Fri Sep 20 22:34:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.59199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:35:09.773 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:35:10.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:09 smithi153 ceph-mon[24571]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:35:10.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:09 smithi115 ceph-mon[23720]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:35:10.039 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:09 smithi115 ceph-mon[28216]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:35:10.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 podman[46286]: 2024-09-20 22:35:09.939763345 +0000 UTC m=+0.023160085 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-09-20T22:35:10.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 podman[46286]: 2024-09-20 22:35:10.039208059 +0000 UTC m=+0.122604779 volume create 9d9217fc33efb587b87e29a9e9a64cea36b7994a620514f67bc3901b9462bae3 2024-09-20T22:35:10.297 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 podman[46286]: 2024-09-20 22:35:10.131312931 +0000 UTC m=+0.214709659 container create 8a0659d27a95c7f534e60c015a85c1cfce7362986d78f27a9dbcf64d524f7e2e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:10.562 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:35:10.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 podman[46286]: 2024-09-20 22:35:10.40033058 +0000 UTC m=+0.483727322 container init 8a0659d27a95c7f534e60c015a85c1cfce7362986d78f27a9dbcf64d524f7e2e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:10.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 podman[46286]: 2024-09-20 22:35:10.403345978 +0000 UTC m=+0.486742698 container start 8a0659d27a95c7f534e60c015a85c1cfce7362986d78f27a9dbcf64d524f7e2e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:10.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[46313]: level=info ts=2024-09-20T22:35:10.414Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-09-20T22:35:10.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[46313]: level=info ts=2024-09-20T22:35:10.414Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-09-20T22:35:10.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[46313]: level=info ts=2024-09-20T22:35:10.415Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.115 port=9094 2024-09-20T22:35:10.655 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[46313]: level=info ts=2024-09-20T22:35:10.416Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-20T22:35:10.655 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[46313]: level=info ts=2024-09-20T22:35:10.440Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T22:35:10.655 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[46313]: level=info ts=2024-09-20T22:35:10.440Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T22:35:10.655 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[46313]: level=info ts=2024-09-20T22:35:10.443Z caller=main.go:518 msg=Listening address=:9093 2024-09-20T22:35:10.655 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[46313]: level=info ts=2024-09-20T22:35:10.443Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-09-20T22:35:10.655 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 bash[46286]: 8a0659d27a95c7f534e60c015a85c1cfce7362986d78f27a9dbcf64d524f7e2e 2024-09-20T22:35:10.655 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:10 smithi115 systemd[1]: Started Ceph alertmanager.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:35:10.938 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-mon[23720]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:10.939 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:10.939 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:10.939 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:10.939 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:35:10.939 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:10.939 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-mon[28216]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:10.939 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:10.939 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:10.939 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:10.939 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:35:10.940 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:10 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:11.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:10 smithi153 ceph-mon[24571]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:11.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:10 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:10 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:10 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:10 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:35:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:10 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:11.616 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:35:11 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:35:11] "GET /metrics HTTP/1.1" 200 191503 "" "Prometheus/2.33.4" 2024-09-20T22:35:11.875 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:11 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:35:11.875 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:11 smithi115 ceph-mon[23720]: Deploying daemon grafana.a on smithi153 2024-09-20T22:35:11.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:11 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:35:11.875 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:11 smithi115 ceph-mon[28216]: Deploying daemon grafana.a on smithi153 2024-09-20T22:35:12.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:11 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:35:12.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:11 smithi153 ceph-mon[24571]: Deploying daemon grafana.a on smithi153 2024-09-20T22:35:12.249 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:35:12.250 INFO:teuthology.orchestra.run.smithi115.stderr:dumped all 2024-09-20T22:35:12.563 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[46313]: level=info ts=2024-09-20T22:35:12.416Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000027741s 2024-09-20T22:35:12.880 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:12 smithi115 ceph-mon[23720]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:12.880 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:12 smithi115 ceph-mon[28216]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:12.903 INFO:teuthology.orchestra.run.smithi115.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-20T22:35:11.674651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48900,"kb_used_data":6148,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912316,"statfs":{"total":767960285184,"available":767910211584,"internally_reserved":0,"allocated":6295552,"data_stored":3510393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002548"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-20T22:33:59.749334+0000","last_change":"2024-09-20T22:33:47.659386+0000","last_active":"2024-09-20T22:33:59.749334+0000","last_peered":"2024-09-20T22:33:59.749334+0000","last_clean":"2024-09-20T22:33:59.749334+0000","last_became_active":"2024-09-20T22:33:47.659042+0000","last_became_peered":"2024-09-20T22:33:47.659042+0000","last_unstale":"2024-09-20T22:33:59.749334+0000","last_undegraded":"2024-09-20T22:33:59.749334+0000","last_fullsized":"2024-09-20T22:33:59.749334+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:31:47.021654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:31:47.021654+0000","last_clean_scrub_stamp":"2024-09-20T22:31:47.021654+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:14:59.291958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6092,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739060,"statfs":{"total":95995035648,"available":95988797440,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.62}]},{"osd":1,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.66800000000000004}]},{"osd":2,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.65800000000000003}]},{"osd":3,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.55200000000000005}]},{"osd":4,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.318},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.436}]},{"osd":5,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.36499999999999999}]},{"osd":6,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.47399999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.34300000000000003}]},{"osd":1,"last update":"Fri Sep 20 22:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.39200000000000002}]},{"osd":2,"last update":"Fri Sep 20 22:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.42999999999999999}]},{"osd":3,"last update":"Fri Sep 20 22:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.375},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.498}]},{"osd":4,"last update":"Fri Sep 20 22:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.53000000000000003}]},{"osd":5,"last update":"Fri Sep 20 22:34:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.56999999999999995}]},{"osd":7,"last update":"Fri Sep 20 22:34:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.59699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:30 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.33800000000000002,"15min":0.317},"min":{"1min":0.22500000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.35199999999999998,"15min":0.32600000000000001},"min":{"1min":0.27500000000000002,"5min":0.192,"15min":0.192},"max":{"1min":0.52200000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.28799999999999998}]},{"osd":2,"last update":"Fri Sep 20 22:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.38600000000000001,"15min":0.37},"min":{"1min":0.29799999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.58699999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.38900000000000001,"15min":0.376},"min":{"1min":0.25700000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.33700000000000002}]},{"osd":3,"last update":"Fri Sep 20 22:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42499999999999999,"15min":0.42899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44800000000000001,"15min":0.45100000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.31}]},{"osd":4,"last update":"Fri Sep 20 22:34:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0029999999999999,"5min":1.718,"15min":1.718},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.434,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88800000000000001,"5min":1.671,"15min":1.671},"last":0.55300000000000005}]},{"osd":5,"last update":"Fri Sep 20 22:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.58599999999999997,"15min":0.57999999999999996},"min":{"1min":0.42999999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.82999999999999996,"5min":0.87,"15min":0.87},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.61099999999999999,"15min":0.60299999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.67400000000000004}]},{"osd":6,"last update":"Fri Sep 20 22:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.60999999999999999}]},{"osd":7,"last update":"Fri Sep 20 22:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.65600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 20 22:34:30 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.33100000000000002,"15min":0.313},"min":{"1min":0.23799999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.35699999999999998,"15min":0.33900000000000002},"min":{"1min":0.221,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.29399999999999998}]},{"osd":2,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.38200000000000001,"15min":0.373},"min":{"1min":0.27100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.45100000000000001,"5min":0.627,"15min":0.627},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.35899999999999999,"15min":0.35399999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.497,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.255}]},{"osd":3,"last update":"Fri Sep 20 22:34:14 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.33300000000000002}]},{"osd":4,"last update":"Fri Sep 20 22:34:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.56799999999999995,"15min":0.57699999999999996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66600000000000004,"5min":1.137,"15min":1.137},"last":0.629},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.61499999999999999,"15min":0.629},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71599999999999997,"5min":1.079,"15min":1.079},"last":0.60299999999999998}]},{"osd":5,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.41899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54300000000000004,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.60999999999999999}]},{"osd":6,"last update":"Fri Sep 20 22:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.67100000000000004}]},{"osd":7,"last update":"Fri Sep 20 22:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.68100000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":239784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.37,"15min":0.36199999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.376},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.375,"15min":0.378},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.52800000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.27500000000000002}]},{"osd":1,"last update":"Fri Sep 20 22:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.39200000000000002,"15min":0.38800000000000001},"min":{"1min":0.28000000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.39100000000000001,"15min":0.39300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.55900000000000005,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.314}]},{"osd":3,"last update":"Fri Sep 20 22:34:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.439,"15min":0.44},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.34499999999999997}]},{"osd":4,"last update":"Fri Sep 20 22:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.67700000000000005,"15min":0.625},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":8.6180000000000003,"5min":8.6180000000000003,"15min":8.6180000000000003},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81799999999999995,"5min":1.194,"15min":1.194},"last":0.60099999999999998}]},{"osd":5,"last update":"Fri Sep 20 22:34:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.41799999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80900000000000005,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40899999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.59099999999999997}]},{"osd":6,"last update":"Fri Sep 20 22:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.67300000000000004}]},{"osd":7,"last update":"Fri Sep 20 22:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.68999999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.35799999999999998,"15min":0.35999999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.372},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.39600000000000002,"15min":0.39500000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.307}]},{"osd":1,"last update":"Fri Sep 20 22:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.433,"15min":0.441},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.34599999999999997}]},{"osd":2,"last update":"Fri Sep 20 22:34:12 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42299999999999999,"15min":0.42199999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.441,"15min":0.44500000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.35799999999999998}]},{"osd":4,"last update":"Fri Sep 20 22:34:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.55400000000000005,"15min":0.56100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.497},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.63100000000000001,"15min":0.64500000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0249999999999999,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":0.52700000000000002}]},{"osd":5,"last update":"Fri Sep 20 22:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.65400000000000003,"15min":0.66000000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.1080000000000001,"5min":1.841,"15min":1.841},"last":0.504}]},{"osd":6,"last update":"Fri Sep 20 22:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.51200000000000001}]},{"osd":7,"last update":"Fri Sep 20 22:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.56000000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986211,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52700000000000002,"15min":0.52600000000000002},"min":{"1min":0.377,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67600000000000005,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.55400000000000005,"15min":0.56599999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65700000000000003,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.70699999999999996}]},{"osd":1,"last update":"Fri Sep 20 22:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53700000000000003,"15min":0.54300000000000004},"min":{"1min":0.36599999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.67900000000000005,"5min":0.877,"15min":0.877},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54500000000000004,"15min":0.55100000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68200000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.65800000000000003}]},{"osd":2,"last update":"Fri Sep 20 22:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52800000000000002,"15min":0.53500000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.58799999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.38300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.73399999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.624}]},{"osd":3,"last update":"Fri Sep 20 22:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.69699999999999995,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.42999999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.70699999999999996,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.69199999999999995}]},{"osd":5,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.442},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.42299999999999999,"15min":0.41899999999999998},"min":{"1min":0.33900000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56899999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.36599999999999999}]},{"osd":6,"last update":"Fri Sep 20 22:34:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.41699999999999998}]},{"osd":7,"last update":"Fri Sep 20 22:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.48899999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5636,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739516,"statfs":{"total":95995035648,"available":95989264384,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.378,"5min":0.372,"15min":0.372},"max":{"1min":0.66600000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.5},"min":{"1min":0.33300000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.39700000000000002}]},{"osd":1,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.68999999999999995,"5min":0.871,"15min":0.871},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.56100000000000005,"15min":0.56699999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.67800000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.434}]},{"osd":2,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.54500000000000004,"15min":0.52000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.8470000000000004,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.39400000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.66700000000000004,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.442},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52900000000000003,"15min":0.52500000000000002},"min":{"1min":0.40500000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73599999999999999,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.499}]},{"osd":4,"last update":"Fri Sep 20 22:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.38900000000000001,"15min":0.377},"min":{"1min":0.27100000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.41599999999999998,"15min":0.40600000000000003},"min":{"1min":0.28100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.67200000000000004,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.40400000000000003}]},{"osd":6,"last update":"Fri Sep 20 22:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.379},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.34200000000000003}]},{"osd":7,"last update":"Fri Sep 20 22:34:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.48399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:35:12.904 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-20T22:35:12.904 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-20T22:35:12.905 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-20T22:35:12.905 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T22:35:13.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:12 smithi153 ceph-mon[24571]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:13.113 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:35:13.869 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:13 smithi115 ceph-mon[23720]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:35:13.870 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:13 smithi115 ceph-mon[28216]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:35:14.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:13 smithi153 ceph-mon[24571]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:35:14.790 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-09-20T22:35:14.790 INFO:teuthology.orchestra.run.smithi115.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-20T22:35:15.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:14 smithi153 ceph-mon[24571]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:15.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:14 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:15.092 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:14 smithi115 ceph-mon[23720]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:15.092 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:14 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:15.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:14 smithi115 ceph-mon[28216]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:15.092 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:14 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:15.365 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-20T22:35:15.366 INFO:tasks.cephadm:Setup complete, yielding 2024-09-20T22:35:15.366 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T22:35:15.383 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi115.front.sepia.ceph.com 2024-09-20T22:35:15.384 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-09-20T22:35:16.012 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:15 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1685420990' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T22:35:16.012 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:15 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1685420990' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T22:35:16.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:15 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1685420990' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T22:35:16.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:16 smithi115 ceph-mon[23720]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:16.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:16 smithi115 ceph-mon[28216]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:17.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:16 smithi153 ceph-mon[24571]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:17.751 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T22:35:17.761 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi115.front.sepia.ceph.com 2024-09-20T22:35:17.761 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-09-20T22:35:18.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:18 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2758238973' entity='client.admin' 2024-09-20T22:35:18.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:18 smithi115 ceph-mon[23720]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:18.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:18 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2758238973' entity='client.admin' 2024-09-20T22:35:18.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:18 smithi115 ceph-mon[28216]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:18.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:18 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2758238973' entity='client.admin' 2024-09-20T22:35:18.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:18 smithi153 ceph-mon[24571]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:19.528 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:35:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:35:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:35:20.672 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:20 smithi153 systemd[1]: Starting Ceph grafana.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:35:20.677 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:20 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[46313]: level=info ts=2024-09-20T22:35:20.419Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.00299556s 2024-09-20T22:35:20.710 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:35:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: "id": "697a2e5f-adc5-4193-9d89-c696b5413264", 2024-09-20T22:35:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: "name": "r", 2024-09-20T22:35:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: "current_period": "0dd5d445-b252-4bfb-8850-bdfd525b81a0", 2024-09-20T22:35:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: "epoch": 1 2024-09-20T22:35:20.711 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:35:20.977 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:20 smithi153 ceph-mon[24571]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T22:35:20.977 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:20 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2807970692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T22:35:20.977 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:20 smithi153 ceph-mon[24571]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:20.978 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:20 smithi153 podman[40667]: 2024-09-20 22:35:20.662248146 +0000 UTC m=+0.015283016 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-09-20T22:35:20.978 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:20 smithi153 podman[40667]: 2024-09-20 22:35:20.782433621 +0000 UTC m=+0.135468465 container create 1549ac2de17eb6f2c8b988043ee10f0427eebdcbf25fdb2f2fcbf764552feeaf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, build-date=2022-03-28T10:36:18.413762, name=ubi8, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, distribution-scope=public, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, maintainer=Paul Cuzner ) 2024-09-20T22:35:20.978 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:20 smithi153 podman[40667]: 2024-09-20 22:35:20.97631027 +0000 UTC m=+0.329345129 container init 1549ac2de17eb6f2c8b988043ee10f0427eebdcbf25fdb2f2fcbf764552feeaf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, vendor=Red Hat, Inc., io.buildah.version=1.24.2, maintainer=Paul Cuzner , release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, io.openshift.expose-services=) 2024-09-20T22:35:21.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:20 smithi115 ceph-mon[23720]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T22:35:21.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:20 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2807970692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T22:35:21.066 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:20 smithi115 ceph-mon[23720]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:21.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:20 smithi115 ceph-mon[28216]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T22:35:21.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:20 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2807970692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T22:35:21.067 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:20 smithi115 ceph-mon[28216]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:21.230 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:20 smithi153 podman[40667]: 2024-09-20 22:35:20.98023617 +0000 UTC m=+0.333271020 container start 1549ac2de17eb6f2c8b988043ee10f0427eebdcbf25fdb2f2fcbf764552feeaf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a, com.redhat.component=ubi8-container, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, version=8.5, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=) 2024-09-20T22:35:21.230 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-09-20T22:35:21.230 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-09-20T22:35:21.230 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-09-20T22:35:21.231 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-09-20T22:35:21.231 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-20T22:35:21.231 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-20T22:35:21.231 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-20T22:35:21.231 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-20T22:35:21.231 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-09-20T22:35:21.231 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-09-20T22:35:21.231 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-09-20T22:35:21.231 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-09-20T22:35:21.232 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-09-20T22:35:21.232 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="App mode production" logger=settings 2024-09-20T22:35:21.232 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-09-20T22:35:21.232 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-20T22:35:21.232 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-09-20T22:35:21.232 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-09-20T22:35:21.232 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-09-20T22:35:21.232 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-09-20T22:35:21.232 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-09-20T22:35:21.232 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-09-20T22:35:21.232 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-09-20T22:35:21.232 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-09-20T22:35:21.233 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-09-20T22:35:21.233 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-09-20T22:35:21.233 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-09-20T22:35:21.233 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-09-20T22:35:21.233 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-09-20T22:35:21.233 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-09-20T22:35:21.233 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-09-20T22:35:21.233 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-09-20T22:35:21.233 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-09-20T22:35:21.233 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-09-20T22:35:21.233 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-09-20T22:35:21.233 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-09-20T22:35:21.233 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-09-20T22:35:21.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-09-20T22:35:21.235 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-09-20T22:35:21.235 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-09-20T22:35:21.235 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-09-20T22:35:21.236 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-09-20T22:35:21.236 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-09-20T22:35:21.236 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-09-20T22:35:21.236 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-09-20T22:35:21.236 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-09-20T22:35:21.236 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-09-20T22:35:21.236 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-09-20T22:35:21.236 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-09-20T22:35:21.236 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-09-20T22:35:21.236 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-09-20T22:35:21.236 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 bash[40667]: 1549ac2de17eb6f2c8b988043ee10f0427eebdcbf25fdb2f2fcbf764552feeaf 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 systemd[1]: Started Ceph grafana.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-09-20T22:35:21.237 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-09-20T22:35:21.238 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-09-20T22:35:21.238 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-09-20T22:35:21.238 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-09-20T22:35:21.238 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-09-20T22:35:21.238 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-09-20T22:35:21.238 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-09-20T22:35:21.238 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-09-20T22:35:21.238 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-09-20T22:35:21.238 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-09-20T22:35:21.238 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-09-20T22:35:21.238 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-09-20T22:35:21.238 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-09-20T22:35:21.239 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-09-20T22:35:21.239 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-09-20T22:35:21.239 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-09-20T22:35:21.239 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-09-20T22:35:21.239 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-09-20T22:35:21.239 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-09-20T22:35:21.239 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-09-20T22:35:21.239 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-20T22:35:21.239 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-09-20T22:35:21.239 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-20T22:35:21.239 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-09-20T22:35:21.239 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-20T22:35:21.239 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-09-20T22:35:21.240 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-09-20T22:35:21.241 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-09-20T22:35:21.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-09-20T22:35:21.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-09-20T22:35:21.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-09-20T22:35:21.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-09-20T22:35:21.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-09-20T22:35:21.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-09-20T22:35:21.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-09-20T22:35:21.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-09-20T22:35:21.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-09-20T22:35:21.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-09-20T22:35:21.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-09-20T22:35:21.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-09-20T22:35:21.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-09-20T22:35:21.243 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-09-20T22:35:21.243 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-09-20T22:35:21.243 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-09-20T22:35:21.243 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-09-20T22:35:21.243 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-09-20T22:35:21.243 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-09-20T22:35:21.243 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-09-20T22:35:21.243 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-09-20T22:35:21.243 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-20T22:35:21.244 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-20T22:35:21.244 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-09-20T22:35:21.244 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-09-20T22:35:21.244 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-09-20T22:35:21.244 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-09-20T22:35:21.244 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-20T22:35:21.244 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-09-20T22:35:21.244 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-20T22:35:21.244 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-09-20T22:35:21.244 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-09-20T22:35:21.244 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-20T22:35:21.244 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-09-20T22:35:21.244 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-09-20T22:35:21.245 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-09-20T22:35:21.245 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-09-20T22:35:21.245 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-09-20T22:35:21.245 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-09-20T22:35:21.245 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-09-20T22:35:21.245 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-09-20T22:35:21.245 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-09-20T22:35:21.245 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-09-20T22:35:21.357 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:35:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:35:21] "GET /metrics HTTP/1.1" 200 191505 "" "Prometheus/2.33.4" 2024-09-20T22:35:21.383 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-09-20T22:35:21.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-09-20T22:35:21.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-09-20T22:35:21.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-09-20T22:35:21.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-09-20T22:35:21.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-09-20T22:35:21.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-09-20T22:35:21.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-09-20T22:35:21.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-09-20T22:35:21.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-09-20T22:35:21.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-09-20T22:35:21.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-09-20T22:35:21.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-20T22:35:21.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-20T22:35:21.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-20T22:35:21.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-09-20T22:35:21.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-20T22:35:21.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-09-20T22:35:21.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-09-20T22:35:21.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-09-20T22:35:21.484 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-09-20T22:35:21.484 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-09-20T22:35:21.484 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-09-20T22:35:21.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-09-20T22:35:21.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-09-20T22:35:21.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-09-20T22:35:21.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-09-20T22:35:21.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-09-20T22:35:21.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-20T22:35:21.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-09-20T22:35:21.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-09-20T22:35:21.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-20T22:35:21.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-09-20T22:35:21.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-09-20T22:35:21.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-09-20T22:35:21.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-09-20T22:35:21.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-09-20T22:35:21.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-09-20T22:35:21.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-09-20T22:35:21.489 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-09-20T22:35:21.490 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-09-20T22:35:21.491 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-09-20T22:35:21.491 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-09-20T22:35:21.491 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-20T22:35:21.491 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-09-20T22:35:21.491 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-09-20T22:35:21.491 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-09-20T22:35:21.491 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-09-20T22:35:21.491 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-09-20T22:35:21.491 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-20T22:35:21.491 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-09-20T22:35:21.491 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-09-20T22:35:21.491 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-09-20T22:35:21.491 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-09-20T22:35:21.492 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-09-20T22:35:21.493 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-09-20T22:35:21.493 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-09-20T22:35:21.493 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-09-20T22:35:21.493 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-09-20T22:35:21.493 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-20T22:35:21.494 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-20T22:35:21.495 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-09-20T22:35:21.495 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-20T22:35:21.495 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-09-20T22:35:21.495 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-09-20T22:35:21.495 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-09-20T22:35:21.495 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-20T22:35:21.495 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-09-20T22:35:21.495 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-09-20T22:35:21.495 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-09-20T22:35:21.495 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-09-20T22:35:21.495 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-20T22:35:21.495 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-09-20T22:35:21.495 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-09-20T22:35:21.496 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-09-20T22:35:21.496 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-09-20T22:35:21.496 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-09-20T22:35:21.496 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-20T22:35:21.496 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-09-20T22:35:21.496 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-09-20T22:35:21.496 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-20T22:35:21.496 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-09-20T22:35:21.496 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-20T22:35:21.496 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-20T22:35:21.497 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-20T22:35:21.497 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-09-20T22:35:21.497 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-09-20T22:35:21.497 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-09-20T22:35:21.497 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-09-20T22:35:21.497 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-09-20T22:35:21.497 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-20T22:35:21.497 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-09-20T22:35:21.497 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-09-20T22:35:21.497 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-09-20T22:35:21.498 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-09-20T22:35:21.498 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-09-20T22:35:21.498 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-09-20T22:35:21.498 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-09-20T22:35:21.498 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-09-20T22:35:21.498 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-09-20T22:35:21.498 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-09-20T22:35:21.498 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-09-20T22:35:21.498 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-09-20T22:35:21.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2807970692' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T22:35:21.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-mon[24571]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T22:35:21.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:21.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:21.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:35:21.780 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-09-20T22:35:21.780 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-09-20T22:35:21.780 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-09-20T22:35:21.780 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-09-20T22:35:21.780 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-09-20T22:35:21.780 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-09-20T22:35:21.780 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-09-20T22:35:21.780 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-09-20T22:35:21.780 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-09-20T22:35:21.780 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-09-20T22:35:21.780 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-09-20T22:35:21.780 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-09-20T22:35:21.780 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-09-20T22:35:21.781 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=537.834333ms 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-09-20T22:35:21.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-09-20T22:35:21.783 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:35:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:35:21+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-09-20T22:35:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:21 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2807970692' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T22:35:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:21 smithi115 ceph-mon[23720]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T22:35:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:21 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:21 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:21 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:35:22.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:21 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2807970692' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T22:35:22.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:21 smithi115 ceph-mon[28216]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T22:35:22.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:21 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:22.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:21 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:22.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:21 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:35:22.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:22 smithi115 ceph-mon[28216]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T22:35:22.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:22 smithi115 ceph-mon[28216]: pgmap v47: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:22.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:22 smithi115 ceph-mon[23720]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T22:35:22.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:22 smithi115 ceph-mon[23720]: pgmap v47: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:22.956 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:22 smithi153 ceph-mon[24571]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T22:35:22.957 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:22 smithi153 ceph-mon[24571]: pgmap v47: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:23.763 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:35:23.763 INFO:teuthology.orchestra.run.smithi115.stdout: "id": "8b128d1d-a255-4e2b-9f94-d0fb76278dca", 2024-09-20T22:35:23.763 INFO:teuthology.orchestra.run.smithi115.stdout: "name": "default", 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: "api_name": "default", 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: "is_master": "true", 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: "endpoints": [], 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: "hostnames": [], 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: "hostnames_s3website": [], 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: "master_zone": "", 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: "zones": [], 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: "placement_targets": [], 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: "default_placement": "", 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: "realm_id": "697a2e5f-adc5-4193-9d89-c696b5413264", 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: "sync_policy": { 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: "groups": [] 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:35:23.764 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:35:24.475 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-09-20T22:35:24.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:24 smithi153 ceph-mon[24571]: pgmap v48: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:25.050 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:24 smithi115 ceph-mon[23720]: pgmap v48: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:25.050 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:24 smithi115 ceph-mon[28216]: pgmap v48: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:25.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:25 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:25.792 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:25 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:25.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:25 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:25.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:25 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:26.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:25 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:26.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:25 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:27.140 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:26 smithi115 ceph-mon[23720]: pgmap v49: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-09-20T22:35:27.140 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:26 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:27.140 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:26 smithi115 ceph-mon[28216]: pgmap v49: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-09-20T22:35:27.140 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:26 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:27.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:26 smithi153 ceph-mon[24571]: pgmap v49: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 681 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-09-20T22:35:27.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:26 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:27.684 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:35:27.684 INFO:teuthology.orchestra.run.smithi115.stdout: "id": "b1606caa-7969-469e-afdf-f4780f8e516e", 2024-09-20T22:35:27.684 INFO:teuthology.orchestra.run.smithi115.stdout: "name": "z", 2024-09-20T22:35:27.684 INFO:teuthology.orchestra.run.smithi115.stdout: "domain_root": "z.rgw.meta:root", 2024-09-20T22:35:27.684 INFO:teuthology.orchestra.run.smithi115.stdout: "control_pool": "z.rgw.control", 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "gc_pool": "z.rgw.log:gc", 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "lc_pool": "z.rgw.log:lc", 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "log_pool": "z.rgw.log", 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "roles_pool": "z.rgw.meta:roles", 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "otp_pool": "z.rgw.otp", 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "system_key": { 2024-09-20T22:35:27.685 INFO:teuthology.orchestra.run.smithi115.stdout: "access_key": "", 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: "secret_key": "" 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: "placement_pools": [ 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: "key": "default-placement", 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: "val": { 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: "index_pool": "z.rgw.buckets.index", 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: "storage_classes": { 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: "STANDARD": { 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: "data_pool": "z.rgw.buckets.data" 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-09-20T22:35:27.686 INFO:teuthology.orchestra.run.smithi115.stdout: "index_type": 0 2024-09-20T22:35:27.687 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:35:27.687 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:35:27.687 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:35:27.687 INFO:teuthology.orchestra.run.smithi115.stdout: "realm_id": "697a2e5f-adc5-4193-9d89-c696b5413264", 2024-09-20T22:35:27.687 INFO:teuthology.orchestra.run.smithi115.stdout: "notif_pool": "z.rgw.log:notif" 2024-09-20T22:35:27.687 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:35:28.244 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-09-20T22:35:28.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:28 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:28.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:28 smithi153 ceph-mon[24571]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T22:35:28.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:28 smithi153 ceph-mon[24571]: Reconfiguring daemon alertmanager.a on smithi115 2024-09-20T22:35:28.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:28 smithi153 ceph-mon[24571]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-09-20T22:35:28.834 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:28 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:28.834 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:28 smithi115 ceph-mon[23720]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T22:35:28.834 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:28 smithi115 ceph-mon[23720]: Reconfiguring daemon alertmanager.a on smithi115 2024-09-20T22:35:28.834 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:28 smithi115 ceph-mon[23720]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-09-20T22:35:28.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:28 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:28.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:28 smithi115 ceph-mon[28216]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T22:35:28.834 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:28 smithi115 ceph-mon[28216]: Reconfiguring daemon alertmanager.a on smithi115 2024-09-20T22:35:28.835 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:28 smithi115 ceph-mon[28216]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-09-20T22:35:29.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:35:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:35:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:35:30.828 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:30 smithi115 ceph-mon[23720]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 455 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-09-20T22:35:30.828 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:30 smithi115 ceph-mon[28216]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 455 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-09-20T22:35:31.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:30 smithi153 ceph-mon[24571]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 455 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-09-20T22:35:31.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:35:31 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:35:31] "GET /metrics HTTP/1.1" 200 193056 "" "Prometheus/2.33.4" 2024-09-20T22:35:32.154 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:32 smithi115 systemd[1]: Stopping Ceph alertmanager.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:35:32.454 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:32 smithi115 bash[47892]: Error: no container with name or ID "ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager.a" found: no such container 2024-09-20T22:35:32.735 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:32 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[46313]: level=info ts=2024-09-20T22:35:32.573Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-09-20T22:35:32.735 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:32 smithi115 podman[47908]: 2024-09-20 22:35:32.593866767 +0000 UTC m=+0.297219835 container died 8a0659d27a95c7f534e60c015a85c1cfce7362986d78f27a9dbcf64d524f7e2e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:33.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:32 smithi153 ceph-mon[24571]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-09-20T22:35:33.075 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:32 smithi115 ceph-mon[23720]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-09-20T22:35:33.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:32 smithi115 ceph-mon[28216]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-09-20T22:35:33.076 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:32 smithi115 podman[47908]: 2024-09-20 22:35:32.906461128 +0000 UTC m=+0.609814188 container cleanup 8a0659d27a95c7f534e60c015a85c1cfce7362986d78f27a9dbcf64d524f7e2e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:33.076 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:32 smithi115 bash[47908]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a 2024-09-20T22:35:33.076 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:32 smithi115 bash[47992]: Error: no container with name or ID "ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager.a" found: no such container 2024-09-20T22:35:33.398 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 podman[47975]: 2024-09-20 22:35:33.074144984 +0000 UTC m=+0.479463192 container remove 8a0659d27a95c7f534e60c015a85c1cfce7362986d78f27a9dbcf64d524f7e2e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:33.398 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 podman[47975]: 2024-09-20 22:35:33.174959383 +0000 UTC m=+0.580277580 volume remove 9d9217fc33efb587b87e29a9e9a64cea36b7994a620514f67bc3901b9462bae3 2024-09-20T22:35:33.398 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-20T22:35:33.398 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 systemd[1]: Stopped Ceph alertmanager.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:35:33.398 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 systemd[1]: Starting Ceph alertmanager.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:35:33.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 podman[48053]: 2024-09-20 22:35:33.299893216 +0000 UTC m=+0.016341724 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-09-20T22:35:33.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 podman[48053]: 2024-09-20 22:35:33.400512138 +0000 UTC m=+0.116960688 volume create 8dbc72113590d272c87a93bbc8d945872930bdd16d18294408c994aebb589098 2024-09-20T22:35:33.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 podman[48053]: 2024-09-20 22:35:33.475844364 +0000 UTC m=+0.192292873 container create cd90350c7e9aca6f9694f121c5a99f930a140b68f823089a1fed95a143fb97b8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:34.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:33 smithi153 ceph-mon[24571]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T22:35:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:33 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2169110766' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T22:35:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:33 smithi153 ceph-mon[24571]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T22:35:34.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-mon[28216]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T22:35:34.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2169110766' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T22:35:34.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-mon[28216]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T22:35:34.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-mon[23720]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T22:35:34.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2169110766' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T22:35:34.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-mon[23720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-20T22:35:34.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 podman[48053]: 2024-09-20 22:35:33.795558138 +0000 UTC m=+0.512006655 container init cd90350c7e9aca6f9694f121c5a99f930a140b68f823089a1fed95a143fb97b8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:34.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 podman[48053]: 2024-09-20 22:35:33.798304741 +0000 UTC m=+0.514753261 container start cd90350c7e9aca6f9694f121c5a99f930a140b68f823089a1fed95a143fb97b8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:34.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=info ts=2024-09-20T22:35:33.808Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-09-20T22:35:34.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=info ts=2024-09-20T22:35:33.808Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-09-20T22:35:34.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=info ts=2024-09-20T22:35:33.809Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.115 port=9094 2024-09-20T22:35:34.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=info ts=2024-09-20T22:35:33.810Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-20T22:35:34.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=info ts=2024-09-20T22:35:33.835Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T22:35:34.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=info ts=2024-09-20T22:35:33.835Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T22:35:34.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=info ts=2024-09-20T22:35:33.838Z caller=main.go:518 msg=Listening address=:9093 2024-09-20T22:35:34.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=info ts=2024-09-20T22:35:33.838Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-09-20T22:35:34.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 bash[48053]: cd90350c7e9aca6f9694f121c5a99f930a140b68f823089a1fed95a143fb97b8 2024-09-20T22:35:34.156 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:33 smithi115 systemd[1]: Started Ceph alertmanager.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:35:35.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:34 smithi153 ceph-mon[24571]: pgmap v54: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-09-20T22:35:35.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:34 smithi153 ceph-mon[24571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-20T22:35:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:34 smithi153 ceph-mon[24571]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T22:35:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:34 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:34 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:35.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:34 smithi115 ceph-mon[23720]: pgmap v54: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-09-20T22:35:35.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:34 smithi115 ceph-mon[23720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-20T22:35:35.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:34 smithi115 ceph-mon[23720]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T22:35:35.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:34 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:35.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:34 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:35.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:34 smithi115 ceph-mon[28216]: pgmap v54: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-09-20T22:35:35.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:34 smithi115 ceph-mon[28216]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-20T22:35:35.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:34 smithi115 ceph-mon[28216]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T22:35:35.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:34 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:35.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:34 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:35.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:35 smithi153 ceph-mon[24571]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T22:35:35.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:35 smithi153 ceph-mon[24571]: Reconfiguring daemon prometheus.a on smithi153 2024-09-20T22:35:35.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:35 smithi153 ceph-mon[24571]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T22:35:35.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:35 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2169110766' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T22:35:35.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:35 smithi153 ceph-mon[24571]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T22:35:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:35 smithi115 ceph-mon[23720]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T22:35:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:35 smithi115 ceph-mon[23720]: Reconfiguring daemon prometheus.a on smithi153 2024-09-20T22:35:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:35 smithi115 ceph-mon[23720]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T22:35:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:35 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2169110766' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T22:35:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:35 smithi115 ceph-mon[23720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T22:35:36.155 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=info ts=2024-09-20T22:35:35.810Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000069195s 2024-09-20T22:35:36.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:35 smithi115 ceph-mon[28216]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T22:35:36.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:35 smithi115 ceph-mon[28216]: Reconfiguring daemon prometheus.a on smithi153 2024-09-20T22:35:36.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:35 smithi115 ceph-mon[28216]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T22:35:36.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:35 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2169110766' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T22:35:36.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:35 smithi115 ceph-mon[28216]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-20T22:35:36.955 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:36 smithi153 ceph-mon[24571]: pgmap v57: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-09-20T22:35:36.956 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:36 smithi153 ceph-mon[24571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-20T22:35:36.956 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:36 smithi153 ceph-mon[24571]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T22:35:37.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:36 smithi115 ceph-mon[23720]: pgmap v57: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-09-20T22:35:37.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:36 smithi115 ceph-mon[23720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-20T22:35:37.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:36 smithi115 ceph-mon[23720]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T22:35:37.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:36 smithi115 ceph-mon[28216]: pgmap v57: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-09-20T22:35:37.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:36 smithi115 ceph-mon[28216]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-20T22:35:37.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:36 smithi115 ceph-mon[28216]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T22:35:37.277 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 systemd[1]: Stopping Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:35:37.277 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 bash[41341]: Error: no container with name or ID "ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus.a" found: no such container 2024-09-20T22:35:37.529 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:35:37.275Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T22:35:37.529 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:35:37.275Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-09-20T22:35:37.529 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:35:37.275Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-09-20T22:35:37.529 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:35:37.275Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-09-20T22:35:37.529 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:35:37.275Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-09-20T22:35:37.529 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:35:37.275Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-09-20T22:35:37.529 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:35:37.276Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T22:35:37.529 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:35:37.276Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-09-20T22:35:37.529 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:35:37.276Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T22:35:37.529 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:35:37.276Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T22:35:37.530 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:35:37.276Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-09-20T22:35:37.530 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[39659]: ts=2024-09-20T22:35:37.276Z caller=main.go:1066 level=info msg="See you next time!" 2024-09-20T22:35:37.530 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 podman[41352]: 2024-09-20 22:35:37.285530383 +0000 UTC m=+0.134524318 container died e01d4cd3b2f5d69e793845aa8313723d96bafb86a611f8c7f63dbb749cc26d55 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:37.860 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-mon[24571]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T22:35:37.860 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:37 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T22:35:37.861 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 podman[41352]: 2024-09-20 22:35:37.649649657 +0000 UTC m=+0.498643577 container cleanup e01d4cd3b2f5d69e793845aa8313723d96bafb86a611f8c7f63dbb749cc26d55 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:37.861 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 bash[41352]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T22:35:38.128 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 podman[41364]: 2024-09-20 22:35:37.859367136 +0000 UTC m=+0.574509418 container remove e01d4cd3b2f5d69e793845aa8313723d96bafb86a611f8c7f63dbb749cc26d55 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:38.128 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 bash[41379]: Error: no container with name or ID "ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus.a" found: no such container 2024-09-20T22:35:38.128 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-20T22:35:38.128 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 systemd[1]: Stopped Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:35:38.128 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:37 smithi153 systemd[1]: Starting Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:35:38.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:37 smithi115 ceph-mon[23720]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T22:35:38.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:37 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T22:35:38.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:37 smithi115 ceph-mon[28216]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T22:35:38.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:37 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T22:35:38.405 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 podman[41443]: 2024-09-20 22:35:38.127157036 +0000 UTC m=+0.107023340 container create 7dda2e0d26909fa42a5383fb311694729e8bf27783e5678d7d8ed1a7f376c662 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:38.405 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 podman[41443]: 2024-09-20 22:35:38.036856659 +0000 UTC m=+0.016722925 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-09-20T22:35:38.405 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 podman[41443]: 2024-09-20 22:35:38.337411706 +0000 UTC m=+0.317277970 container init 7dda2e0d26909fa42a5383fb311694729e8bf27783e5678d7d8ed1a7f376c662 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 podman[41443]: 2024-09-20 22:35:38.340176896 +0000 UTC m=+0.320043163 container start 7dda2e0d26909fa42a5383fb311694729e8bf27783e5678d7d8ed1a7f376c662 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.363Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.363Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.363Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.363Z caller=main.go:518 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi153 (none))" 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.363Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.363Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.365Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.365Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.367Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.369Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.369Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.624µs 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.369Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T22:35:38.406 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 bash[41443]: 7dda2e0d26909fa42a5383fb311694729e8bf27783e5678d7d8ed1a7f376c662 2024-09-20T22:35:38.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 systemd[1]: Started Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:35:38.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.570Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-20T22:35:38.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.571Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-20T22:35:38.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.571Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=38.337µs wal_replay_duration=202.196724ms total_replay_duration=202.248488ms 2024-09-20T22:35:38.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.574Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T22:35:38.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.574Z caller=main.go:947 level=info msg="TSDB started" 2024-09-20T22:35:38.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.574Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T22:35:38.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.592Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.134758ms db_storage=781ns remote_storage=2.097µs web_handler=440ns query_engine=713ns scrape=518.979µs scrape_sd=36.949µs notify=29.001µs notify_sd=11.689µs rules=17.199718ms 2024-09-20T22:35:38.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:35:38.592Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-09-20T22:35:39.148 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: pgmap v60: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-09-20T22:35:39.148 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-20T22:35:39.148 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T22:35:39.148 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T22:35:39.148 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:39.148 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:35:39.148 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.115:9093"}]: dispatch 2024-09-20T22:35:39.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:39.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:35:39.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.153:9095"}]: dispatch 2024-09-20T22:35:39.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:39.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:35:39.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.153:3000"}]: dispatch 2024-09-20T22:35:39.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:39.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:39.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:35:39.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T22:35:39.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T22:35:39.149 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-20T22:35:39.149 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: pgmap v60: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.115:9093"}]: dispatch 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.153:9095"}]: dispatch 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.153:3000"}]: dispatch 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:39.150 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:39.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:35:39.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T22:35:39.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T22:35:39.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:38 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-20T22:35:39.158 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: pgmap v60: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-09-20T22:35:39.158 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-20T22:35:39.158 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T22:35:39.158 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T22:35:39.158 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:39.158 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:35:39.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.115:9093"}]: dispatch 2024-09-20T22:35:39.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:39.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:35:39.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.153:9095"}]: dispatch 2024-09-20T22:35:39.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:39.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:35:39.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.153:3000"}]: dispatch 2024-09-20T22:35:39.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:39.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:39.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:35:39.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T22:35:39.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T22:35:39.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:38 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-20T22:35:39.831 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:35:39.831 INFO:teuthology.orchestra.run.smithi115.stdout: "id": "03e148d1-5bfb-4c3d-b040-a6f444b6f5fd", 2024-09-20T22:35:39.831 INFO:teuthology.orchestra.run.smithi115.stdout: "epoch": 1, 2024-09-20T22:35:39.831 INFO:teuthology.orchestra.run.smithi115.stdout: "predecessor_uuid": "0dd5d445-b252-4bfb-8850-bdfd525b81a0", 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: "sync_status": [], 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: "period_map": { 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: "id": "03e148d1-5bfb-4c3d-b040-a6f444b6f5fd", 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: "zonegroups": [ 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: "id": "8b128d1d-a255-4e2b-9f94-d0fb76278dca", 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: "name": "default", 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: "api_name": "default", 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: "is_master": "true", 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: "endpoints": [], 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: "hostnames": [], 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: "hostnames_s3website": [], 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: "master_zone": "b1606caa-7969-469e-afdf-f4780f8e516e", 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: "zones": [ 2024-09-20T22:35:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: "id": "b1606caa-7969-469e-afdf-f4780f8e516e", 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: "name": "z", 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: "endpoints": [], 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: "log_meta": "false", 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: "log_data": "false", 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: "bucket_index_max_shards": 11, 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: "read_only": "false", 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: "tier_type": "", 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: "sync_from_all": "true", 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: "sync_from": [], 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: "redirect_zone": "" 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: "placement_targets": [ 2024-09-20T22:35:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: "name": "default-placement", 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: "tags": [], 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: "storage_classes": [ 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: "STANDARD" 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: "default_placement": "default-placement", 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: "realm_id": "697a2e5f-adc5-4193-9d89-c696b5413264", 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: "sync_policy": { 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: "groups": [] 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:35:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: "short_zone_ids": [ 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: "key": "b1606caa-7969-469e-afdf-f4780f8e516e", 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: "val": 2767451305 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: "master_zonegroup": "8b128d1d-a255-4e2b-9f94-d0fb76278dca", 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: "master_zone": "b1606caa-7969-469e-afdf-f4780f8e516e", 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: "period_config": { 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: "bucket_quota": { 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: "enabled": false, 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: "check_on_raw": false, 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: "max_size": -1, 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: "max_size_kb": 0, 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: "max_objects": -1 2024-09-20T22:35:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "user_quota": { 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "enabled": false, 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "check_on_raw": false, 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "max_size": -1, 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "max_size_kb": 0, 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "max_objects": -1 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "user_ratelimit": { 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "max_read_ops": 0, 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "max_write_ops": 0, 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "max_read_bytes": 0, 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "max_write_bytes": 0, 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "enabled": false 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "bucket_ratelimit": { 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "max_read_ops": 0, 2024-09-20T22:35:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: "max_write_ops": 0, 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: "max_read_bytes": 0, 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: "max_write_bytes": 0, 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: "enabled": false 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: "anonymous_ratelimit": { 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: "max_read_ops": 0, 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: "max_write_ops": 0, 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: "max_read_bytes": 0, 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: "max_write_bytes": 0, 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: "enabled": false 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: "realm_id": "697a2e5f-adc5-4193-9d89-c696b5413264", 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: "realm_name": "r", 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: "realm_epoch": 2 2024-09-20T22:35:39.837 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:35:40.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:35:40.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.115:9093"}]: dispatch 2024-09-20T22:35:40.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:35:40.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.153:9095"}]: dispatch 2024-09-20T22:35:40.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:35:40.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.153:3000"}]: dispatch 2024-09-20T22:35:40.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-09-20T22:35:40.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[23720]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T22:35:40.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:35:40.062 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.115:9093"}]: dispatch 2024-09-20T22:35:40.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:35:40.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.153:9095"}]: dispatch 2024-09-20T22:35:40.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:35:40.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.153:3000"}]: dispatch 2024-09-20T22:35:40.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-09-20T22:35:40.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:39 smithi115 ceph-mon[28216]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T22:35:40.086 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:39 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:35:40.086 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:39 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.115:9093"}]: dispatch 2024-09-20T22:35:40.086 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:39 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:35:40.086 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:39 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.153:9095"}]: dispatch 2024-09-20T22:35:40.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:39 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:35:40.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:39 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.153:3000"}]: dispatch 2024-09-20T22:35:40.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:39 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2020658445' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-09-20T22:35:40.087 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:39 smithi153 ceph-mon[24571]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T22:35:40.706 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-09-20T22:35:40.906 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:40 smithi153 ceph-mon[24571]: pgmap v63: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:41.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:40 smithi115 ceph-mon[23720]: pgmap v63: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:41.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:40 smithi115 ceph-mon[28216]: pgmap v63: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:35:43.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:42 smithi153 ceph-mon[24571]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-09-20T22:35:43.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:42 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:43.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:42 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:42 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:42 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:42 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:35:43.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:42 smithi115 ceph-mon[23720]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-09-20T22:35:43.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:42 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:43.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:42 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:43.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:42 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:43.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:42 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:43.052 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:42 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:35:43.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:42 smithi115 ceph-mon[28216]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-09-20T22:35:43.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:42 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:43.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:42 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:43.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:42 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:43.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:42 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:43.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:42 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:35:43.462 INFO:teuthology.orchestra.run.smithi115.stdout:Scheduled rgw.foo update... 2024-09-20T22:35:44.154 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:35:43 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=info ts=2024-09-20T22:35:43.812Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002357519s 2024-09-20T22:35:44.313 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph orch apply rgw smpl' 2024-09-20T22:35:44.619 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:44 smithi115 ceph-mon[23720]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:44.620 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:44 smithi115 ceph-mon[23720]: Saving service rgw.foo spec with placement count:2 2024-09-20T22:35:44.620 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:44 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:44.620 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:44 smithi115 ceph-mon[23720]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-20T22:35:44.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:44 smithi115 ceph-mon[28216]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:44.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:44 smithi115 ceph-mon[28216]: Saving service rgw.foo spec with placement count:2 2024-09-20T22:35:44.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:44 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:44.620 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:44 smithi115 ceph-mon[28216]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-20T22:35:44.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:44 smithi153 ceph-mon[24571]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:44.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:44 smithi153 ceph-mon[24571]: Saving service rgw.foo spec with placement count:2 2024-09-20T22:35:44.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:44 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:44.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:44 smithi153 ceph-mon[24571]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-20T22:35:46.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[23720]: Saving service rgw.foo spec with placement count:2 2024-09-20T22:35:46.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:46.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:46.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:35:46.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:46.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[23720]: Deploying daemon rgw.foo.smithi115.cotebr on smithi115 2024-09-20T22:35:46.504 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[23720]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-20T22:35:46.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[28216]: Saving service rgw.foo spec with placement count:2 2024-09-20T22:35:46.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:46.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:46.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:35:46.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:46.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[28216]: Deploying daemon rgw.foo.smithi115.cotebr on smithi115 2024-09-20T22:35:46.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:46 smithi115 ceph-mon[28216]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-20T22:35:46.507 INFO:teuthology.orchestra.run.smithi115.stdout:Scheduled rgw.smpl update... 2024-09-20T22:35:46.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:46 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:46 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:46 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:46 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:46 smithi153 ceph-mon[24571]: Saving service rgw.foo spec with placement count:2 2024-09-20T22:35:46.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:46 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:46 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:46.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:46 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:46.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:46 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:35:46.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:46 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:46.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:46 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:46.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:46 smithi153 ceph-mon[24571]: Deploying daemon rgw.foo.smithi115.cotebr on smithi115 2024-09-20T22:35:46.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:46 smithi153 ceph-mon[24571]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-20T22:35:47.456 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph osd pool create foo' 2024-09-20T22:35:47.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:47 smithi115 ceph-mon[23720]: from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:47.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:47 smithi115 ceph-mon[23720]: Saving service rgw.smpl spec with placement count:2 2024-09-20T22:35:47.752 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:47 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:47.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:47 smithi115 ceph-mon[28216]: from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:47.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:47 smithi115 ceph-mon[28216]: Saving service rgw.smpl spec with placement count:2 2024-09-20T22:35:47.752 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:47 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:47.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:47 smithi153 ceph-mon[24571]: from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:47.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:47 smithi153 ceph-mon[24571]: Saving service rgw.smpl spec with placement count:2 2024-09-20T22:35:47.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:47 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:48.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:48 smithi115 ceph-mon[23720]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 1.0 KiB/s wr, 2 op/s 2024-09-20T22:35:48.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:48 smithi115 ceph-mon[28216]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 1.0 KiB/s wr, 2 op/s 2024-09-20T22:35:48.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:48 smithi153 ceph-mon[24571]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 1.0 KiB/s wr, 2 op/s 2024-09-20T22:35:49.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:35:49 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:35:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:35:50.039 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'foo' created 2024-09-20T22:35:50.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:50 smithi115 ceph-mon[23720]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 819 B/s rd, 921 B/s wr, 2 op/s 2024-09-20T22:35:50.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:50 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2504607232' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-20T22:35:50.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:50 smithi115 ceph-mon[28216]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 819 B/s rd, 921 B/s wr, 2 op/s 2024-09-20T22:35:50.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:50 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2504607232' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-20T22:35:50.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:50 smithi153 ceph-mon[24571]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 819 B/s rd, 921 B/s wr, 2 op/s 2024-09-20T22:35:50.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:50 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2504607232' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-20T22:35:50.775 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'rbd pool init foo' 2024-09-20T22:35:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2504607232' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-20T22:35:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[23720]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T22:35:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:35:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[23720]: Deploying daemon rgw.foo.smithi153.vetrnf on smithi153 2024-09-20T22:35:51.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[23720]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T22:35:51.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:51 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2504607232' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-20T22:35:51.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:51 smithi153 ceph-mon[24571]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T22:35:51.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:51 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:51.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:51 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:51.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:51 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:51.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:51 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:35:51.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:51 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:51.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:51 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:51.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:51 smithi153 ceph-mon[24571]: Deploying daemon rgw.foo.smithi153.vetrnf on smithi153 2024-09-20T22:35:51.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:51 smithi153 ceph-mon[24571]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T22:35:51.313 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:35:51] "GET /metrics HTTP/1.1" 200 196874 "" "Prometheus/2.33.4" 2024-09-20T22:35:51.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2504607232' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-20T22:35:51.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[28216]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T22:35:51.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:51.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:51.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:51.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:35:51.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:51.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:51.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[28216]: Deploying daemon rgw.foo.smithi153.vetrnf on smithi153 2024-09-20T22:35:51.314 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:51 smithi115 ceph-mon[28216]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T22:35:52.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:52 smithi153 ceph-mon[24571]: pgmap v72: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.1 KiB/s wr, 140 op/s 2024-09-20T22:35:52.313 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:52 smithi115 ceph-mon[23720]: pgmap v72: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.1 KiB/s wr, 140 op/s 2024-09-20T22:35:52.313 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:52 smithi115 ceph-mon[28216]: pgmap v72: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.1 KiB/s wr, 140 op/s 2024-09-20T22:35:53.363 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:53 smithi115 ceph-mon[23720]: osdmap e74: 8 total, 8 up, 8 in 2024-09-20T22:35:53.364 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:53 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3649140630' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-20T22:35:53.364 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:53 smithi115 ceph-mon[23720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-20T22:35:53.364 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:53 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:53.364 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:53 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:53.364 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:53 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:35:53.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:53 smithi115 ceph-mon[28216]: osdmap e74: 8 total, 8 up, 8 in 2024-09-20T22:35:53.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:53 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3649140630' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-20T22:35:53.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:53 smithi115 ceph-mon[28216]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-20T22:35:53.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:53 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:53.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:53 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:53.364 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:53 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:35:53.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:53 smithi153 ceph-mon[24571]: osdmap e74: 8 total, 8 up, 8 in 2024-09-20T22:35:53.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:53 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3649140630' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-20T22:35:53.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:53 smithi153 ceph-mon[24571]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-20T22:35:53.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:53 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:53.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:53 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:53.369 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:53 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:35:54.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:54 smithi115 ceph-mon[23720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-20T22:35:54.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:54 smithi115 ceph-mon[23720]: osdmap e75: 8 total, 8 up, 8 in 2024-09-20T22:35:54.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:54 smithi115 ceph-mon[23720]: pgmap v75: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 8.2 KiB/s wr, 280 op/s 2024-09-20T22:35:54.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:54 smithi115 ceph-mon[28216]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-20T22:35:54.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:54 smithi115 ceph-mon[28216]: osdmap e75: 8 total, 8 up, 8 in 2024-09-20T22:35:54.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:54 smithi115 ceph-mon[28216]: pgmap v75: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 8.2 KiB/s wr, 280 op/s 2024-09-20T22:35:54.407 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:54 smithi153 ceph-mon[24571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-20T22:35:54.407 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:54 smithi153 ceph-mon[24571]: osdmap e75: 8 total, 8 up, 8 in 2024-09-20T22:35:54.407 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:54 smithi153 ceph-mon[24571]: pgmap v75: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 8.2 KiB/s wr, 280 op/s 2024-09-20T22:35:55.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:55 smithi153 ceph-mon[24571]: osdmap e76: 8 total, 8 up, 8 in 2024-09-20T22:35:55.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:55 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:55.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:55 smithi115 ceph-mon[23720]: osdmap e76: 8 total, 8 up, 8 in 2024-09-20T22:35:55.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:55 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:55.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:55 smithi115 ceph-mon[28216]: osdmap e76: 8 total, 8 up, 8 in 2024-09-20T22:35:55.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:55 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:55.874 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph orch apply iscsi foo u p' 2024-09-20T22:35:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:56 smithi115 ceph-mon[23720]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T22:35:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:56 smithi115 ceph-mon[23720]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 162 KiB/s rd, 767 B/s wr, 250 op/s 2024-09-20T22:35:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:56 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:56.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:56 smithi115 ceph-mon[28216]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T22:35:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:56 smithi115 ceph-mon[28216]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 162 KiB/s rd, 767 B/s wr, 250 op/s 2024-09-20T22:35:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:56 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:56.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:56 smithi153 ceph-mon[24571]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T22:35:56.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:56 smithi153 ceph-mon[24571]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 162 KiB/s rd, 767 B/s wr, 250 op/s 2024-09-20T22:35:56.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:56 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[23720]: Checking dashboard <-> RGW credentials 2024-09-20T22:35:57.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[23720]: Saving service rgw.smpl spec with placement count:2 2024-09-20T22:35:57.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:57.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:57.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:35:57.353 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:57.354 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[23720]: Deploying daemon rgw.smpl.smithi115.pccgis on smithi115 2024-09-20T22:35:57.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[28216]: Checking dashboard <-> RGW credentials 2024-09-20T22:35:57.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[28216]: Saving service rgw.smpl spec with placement count:2 2024-09-20T22:35:57.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:57.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:57.354 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:35:57.355 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.355 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:57.355 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:57 smithi115 ceph-mon[28216]: Deploying daemon rgw.smpl.smithi115.pccgis on smithi115 2024-09-20T22:35:57.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:57 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:57 smithi153 ceph-mon[24571]: Checking dashboard <-> RGW credentials 2024-09-20T22:35:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:57 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:57 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:57 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:57 smithi153 ceph-mon[24571]: Saving service rgw.smpl spec with placement count:2 2024-09-20T22:35:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:57 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:57 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:57 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:35:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:57 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:35:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:57 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:57 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:35:57.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:57 smithi153 ceph-mon[24571]: Deploying daemon rgw.smpl.smithi115.pccgis on smithi115 2024-09-20T22:35:57.747 INFO:teuthology.orchestra.run.smithi115.stdout:Scheduled iscsi.foo update... 2024-09-20T22:35:58.534 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'sleep 120' 2024-09-20T22:35:58.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:58 smithi115 ceph-mon[23720]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 545 B/s wr, 178 op/s 2024-09-20T22:35:58.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:58 smithi115 ceph-mon[23720]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:58.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:58 smithi115 ceph-mon[23720]: Saving service iscsi.foo spec with placement count:1 2024-09-20T22:35:58.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:58 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:58.787 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:58 smithi115 ceph-mon[23720]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T22:35:58.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:58 smithi115 ceph-mon[28216]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 545 B/s wr, 178 op/s 2024-09-20T22:35:58.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:58 smithi115 ceph-mon[28216]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:58.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:58 smithi115 ceph-mon[28216]: Saving service iscsi.foo spec with placement count:1 2024-09-20T22:35:58.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:58 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:58.787 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:58 smithi115 ceph-mon[28216]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T22:35:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:58 smithi153 ceph-mon[24571]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 545 B/s wr, 178 op/s 2024-09-20T22:35:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:58 smithi153 ceph-mon[24571]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:35:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:58 smithi153 ceph-mon[24571]: Saving service iscsi.foo spec with placement count:1 2024-09-20T22:35:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:58 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:35:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:58 smithi153 ceph-mon[24571]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T22:35:59.751 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:35:59 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:35:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:36:00.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:59 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-09-20T22:36:00.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:59 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-09-20T22:36:00.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:59 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-09-20T22:36:00.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:59 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-20T22:36:00.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:59 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-20T22:36:00.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:59 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-09-20T22:36:00.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:59 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-20T22:36:00.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:59 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-20T22:36:00.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:59 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:36:00.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:35:59 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:36:00.074 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-09-20T22:36:00.074 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-09-20T22:36:00.074 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-09-20T22:36:00.074 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-20T22:36:00.075 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-20T22:36:00.075 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-09-20T22:36:00.075 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-20T22:36:00.075 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-20T22:36:00.075 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:36:00.075 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:36:00.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-09-20T22:36:00.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-09-20T22:36:00.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-09-20T22:36:00.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-20T22:36:00.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-20T22:36:00.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-09-20T22:36:00.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-20T22:36:00.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-20T22:36:00.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:36:00.076 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:35:59 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:36:01.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:00 smithi153 ceph-mon[24571]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 511 B/s wr, 167 op/s 2024-09-20T22:36:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:36:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:00 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-09-20T22:36:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:00 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]': finished 2024-09-20T22:36:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:00 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-20T22:36:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:00 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-20T22:36:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:00 smithi153 ceph-mon[24571]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T22:36:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:00 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:36:01.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[23720]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 511 B/s wr, 167 op/s 2024-09-20T22:36:01.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:36:01.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-09-20T22:36:01.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]': finished 2024-09-20T22:36:01.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-20T22:36:01.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-20T22:36:01.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[23720]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T22:36:01.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:36:01.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[28216]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 511 B/s wr, 167 op/s 2024-09-20T22:36:01.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:36:01.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-09-20T22:36:01.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]': finished 2024-09-20T22:36:01.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-20T22:36:01.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-20T22:36:01.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[28216]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T22:36:01.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:00 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:36:01.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:36:01 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:36:01] "GET /metrics HTTP/1.1" 200 207972 "" "Prometheus/2.33.4" 2024-09-20T22:36:02.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:01 smithi153 ceph-mon[24571]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T22:36:02.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:01 smithi115 ceph-mon[23720]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T22:36:02.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:01 smithi115 ceph-mon[28216]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T22:36:02.890 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:02 smithi153 ceph-mon[24571]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-09-20T22:36:02.891 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:02 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:02.891 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:02 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:36:02.891 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:02 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:36:02.891 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:02 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:36:02.891 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:02 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:02.891 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:02 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:02.891 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:02 smithi153 ceph-mon[24571]: Deploying daemon rgw.smpl.smithi153.oznbtc on smithi153 2024-09-20T22:36:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[23720]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-09-20T22:36:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:36:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:36:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:36:03.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:03.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:03.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[23720]: Deploying daemon rgw.smpl.smithi153.oznbtc on smithi153 2024-09-20T22:36:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[28216]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-09-20T22:36:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:36:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:36:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:36:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:02 smithi115 ceph-mon[28216]: Deploying daemon rgw.smpl.smithi153.oznbtc on smithi153 2024-09-20T22:36:04.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:04 smithi153 ceph-mon[24571]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-09-20T22:36:04.951 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:04 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:04.951 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:04 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:04.951 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:04 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:36:04.978 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:04 smithi115 ceph-mon[23720]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-09-20T22:36:04.978 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:04 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:04.978 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:04 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:04.978 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:04 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:36:04.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:04 smithi115 ceph-mon[28216]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-09-20T22:36:04.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:04 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:04.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:04 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:04.978 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:04 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:36:06.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:05 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:06.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:05 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:06.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:05 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:07.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:06 smithi153 ceph-mon[24571]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 583 B/s wr, 293 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:36:07.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:06 smithi115 ceph-mon[23720]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 583 B/s wr, 293 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:36:07.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:06 smithi115 ceph-mon[28216]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 583 B/s wr, 293 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:36:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:08 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:08 smithi153 ceph-mon[24571]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 511 B/s wr, 257 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:36:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:08 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:08 smithi153 ceph-mon[24571]: Checking dashboard <-> RGW credentials 2024-09-20T22:36:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:08 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:08 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:36:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:08 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:36:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:08 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T22:36:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:08 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[23720]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 511 B/s wr, 257 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:36:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[23720]: Checking dashboard <-> RGW credentials 2024-09-20T22:36:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:36:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:36:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T22:36:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[28216]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 511 B/s wr, 257 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:36:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[28216]: Checking dashboard <-> RGW credentials 2024-09-20T22:36:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:36:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:36:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T22:36:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:08 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:09.675 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:36:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:36:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:36:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:09 smithi115 ceph-mon[23720]: Checking pool "foo" exists for service iscsi.foo 2024-09-20T22:36:09.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:09 smithi115 ceph-mon[23720]: Deploying daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:36:09.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:09 smithi115 ceph-mon[28216]: Checking pool "foo" exists for service iscsi.foo 2024-09-20T22:36:09.797 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:09 smithi115 ceph-mon[28216]: Deploying daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:36:10.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:09 smithi153 ceph-mon[24571]: Checking pool "foo" exists for service iscsi.foo 2024-09-20T22:36:10.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:09 smithi153 ceph-mon[24571]: Deploying daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:36:10.732 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:10 smithi115 ceph-mon[23720]: mgrmap e20: y(active, since 2m), standbys: x 2024-09-20T22:36:10.732 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:10 smithi115 ceph-mon[23720]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 412 B/s wr, 207 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:36:10.732 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:10 smithi115 ceph-mon[28216]: mgrmap e20: y(active, since 2m), standbys: x 2024-09-20T22:36:10.732 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:10 smithi115 ceph-mon[28216]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 412 B/s wr, 207 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:36:11.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:10 smithi153 ceph-mon[24571]: mgrmap e20: y(active, since 2m), standbys: x 2024-09-20T22:36:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:10 smithi153 ceph-mon[24571]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s rd, 412 B/s wr, 207 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:36:11.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:36:11 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:36:11] "GET /metrics HTTP/1.1" 200 207972 "" "Prometheus/2.33.4" 2024-09-20T22:36:12.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:12.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:12.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:36:12.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3475531499' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:36:12.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[23720]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 KiB/s rd, 187 B/s wr, 187 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:36:12.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/992068801' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3265629470"}]: dispatch 2024-09-20T22:36:12.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3265629470"}]: dispatch 2024-09-20T22:36:12.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:12.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:12.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:36:12.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3475531499' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:36:12.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[28216]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 KiB/s rd, 187 B/s wr, 187 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:36:12.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/992068801' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3265629470"}]: dispatch 2024-09-20T22:36:12.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:12 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3265629470"}]: dispatch 2024-09-20T22:36:12.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:12 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:12.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:12 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:12.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:12 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:36:12.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:12 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3475531499' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:36:12.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:12 smithi153 ceph-mon[24571]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 122 KiB/s rd, 187 B/s wr, 187 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:36:12.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:12 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/992068801' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3265629470"}]: dispatch 2024-09-20T22:36:12.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:12 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3265629470"}]: dispatch 2024-09-20T22:36:13.344 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:13 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3265629470"}]': finished 2024-09-20T22:36:13.344 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:13 smithi115 ceph-mon[23720]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T22:36:13.344 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:13 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/4099195971' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2244521742"}]: dispatch 2024-09-20T22:36:13.344 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:13 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2244521742"}]: dispatch 2024-09-20T22:36:13.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:13 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3265629470"}]': finished 2024-09-20T22:36:13.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:13 smithi115 ceph-mon[28216]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T22:36:13.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:13 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/4099195971' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2244521742"}]: dispatch 2024-09-20T22:36:13.345 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:13 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2244521742"}]: dispatch 2024-09-20T22:36:13.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:13 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3265629470"}]': finished 2024-09-20T22:36:13.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:13 smithi153 ceph-mon[24571]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T22:36:13.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:13 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/4099195971' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2244521742"}]: dispatch 2024-09-20T22:36:13.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:13 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2244521742"}]: dispatch 2024-09-20T22:36:13.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:36:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:36:13.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2024-09-20T22:36:13.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:36:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:36:13.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-09-20T22:36:14.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:14 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2244521742"}]': finished 2024-09-20T22:36:14.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:14 smithi115 ceph-mon[28216]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T22:36:14.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:14 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/4066896721' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2869410039"}]: dispatch 2024-09-20T22:36:14.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:14 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2869410039"}]: dispatch 2024-09-20T22:36:14.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:14 smithi115 ceph-mon[28216]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 4 op/s 2024-09-20T22:36:14.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:14 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2244521742"}]': finished 2024-09-20T22:36:14.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:14 smithi115 ceph-mon[23720]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T22:36:14.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:14 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/4066896721' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2869410039"}]: dispatch 2024-09-20T22:36:14.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:14 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2869410039"}]: dispatch 2024-09-20T22:36:14.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:14 smithi115 ceph-mon[23720]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 4 op/s 2024-09-20T22:36:14.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:14 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2244521742"}]': finished 2024-09-20T22:36:14.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:14 smithi153 ceph-mon[24571]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T22:36:14.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:14 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/4066896721' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2869410039"}]: dispatch 2024-09-20T22:36:14.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:14 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2869410039"}]: dispatch 2024-09-20T22:36:14.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:14 smithi153 ceph-mon[24571]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 4 op/s 2024-09-20T22:36:15.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2869410039"}]': finished 2024-09-20T22:36:15.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T22:36:15.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3930551939' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/4280258613"}]: dispatch 2024-09-20T22:36:15.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:36:15.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:36:15.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: Adding iSCSI gateway http://:@172.21.15.115:5000 to Dashboard 2024-09-20T22:36:15.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:36:15.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:36:15.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi115"}]: dispatch 2024-09-20T22:36:15.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi115"}]: dispatch 2024-09-20T22:36:15.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:15.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:36:15.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: Checking dashboard <-> RGW credentials 2024-09-20T22:36:15.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2869410039"}]': finished 2024-09-20T22:36:15.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T22:36:15.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3930551939' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/4280258613"}]: dispatch 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: Adding iSCSI gateway http://:@172.21.15.115:5000 to Dashboard 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi115"}]: dispatch 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi115"}]: dispatch 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:36:15.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: Checking dashboard <-> RGW credentials 2024-09-20T22:36:15.408 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.408 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:15 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2869410039"}]': finished 2024-09-20T22:36:15.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T22:36:15.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3930551939' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/4280258613"}]: dispatch 2024-09-20T22:36:15.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:36:15.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:36:15.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: Adding iSCSI gateway http://:@172.21.15.115:5000 to Dashboard 2024-09-20T22:36:15.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:36:15.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:36:15.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi115"}]: dispatch 2024-09-20T22:36:15.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi115"}]: dispatch 2024-09-20T22:36:15.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:15.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:36:15.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: Checking dashboard <-> RGW credentials 2024-09-20T22:36:15.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:15.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:15 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:36:16.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:16 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3930551939' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/4280258613"}]': finished 2024-09-20T22:36:16.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:16 smithi115 ceph-mon[23720]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T22:36:16.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:16 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3457403087' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2928344409"}]: dispatch 2024-09-20T22:36:16.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:16 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2928344409"}]: dispatch 2024-09-20T22:36:16.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:16 smithi115 ceph-mon[23720]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-20T22:36:16.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:16 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3930551939' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/4280258613"}]': finished 2024-09-20T22:36:16.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:16 smithi115 ceph-mon[28216]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T22:36:16.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:16 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3457403087' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2928344409"}]: dispatch 2024-09-20T22:36:16.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:16 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2928344409"}]: dispatch 2024-09-20T22:36:16.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:16 smithi115 ceph-mon[28216]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-20T22:36:16.430 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:16 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3930551939' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/4280258613"}]': finished 2024-09-20T22:36:16.430 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:16 smithi153 ceph-mon[24571]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T22:36:16.430 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:16 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3457403087' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2928344409"}]: dispatch 2024-09-20T22:36:16.430 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:16 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2928344409"}]: dispatch 2024-09-20T22:36:16.430 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:16 smithi153 ceph-mon[24571]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-20T22:36:17.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:17 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2928344409"}]': finished 2024-09-20T22:36:17.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:17 smithi153 ceph-mon[24571]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T22:36:17.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:17 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/11480812' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3858670382"}]: dispatch 2024-09-20T22:36:17.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:17 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3858670382"}]: dispatch 2024-09-20T22:36:17.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:17 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2928344409"}]': finished 2024-09-20T22:36:17.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:17 smithi115 ceph-mon[23720]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T22:36:17.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:17 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/11480812' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3858670382"}]: dispatch 2024-09-20T22:36:17.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:17 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3858670382"}]: dispatch 2024-09-20T22:36:17.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:17 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2928344409"}]': finished 2024-09-20T22:36:17.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:17 smithi115 ceph-mon[28216]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T22:36:17.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:17 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/11480812' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3858670382"}]: dispatch 2024-09-20T22:36:17.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:17 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3858670382"}]: dispatch 2024-09-20T22:36:18.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:18 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3858670382"}]': finished 2024-09-20T22:36:18.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:18 smithi115 ceph-mon[23720]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T22:36:18.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:18 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/203424562' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/3265629470"}]: dispatch 2024-09-20T22:36:18.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:18 smithi115 ceph-mon[23720]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-20T22:36:18.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:18 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3858670382"}]': finished 2024-09-20T22:36:18.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:18 smithi115 ceph-mon[28216]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T22:36:18.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:18 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/203424562' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/3265629470"}]: dispatch 2024-09-20T22:36:18.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:18 smithi115 ceph-mon[28216]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-20T22:36:18.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:18 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3858670382"}]': finished 2024-09-20T22:36:18.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:18 smithi153 ceph-mon[24571]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T22:36:18.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:18 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/203424562' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/3265629470"}]: dispatch 2024-09-20T22:36:18.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:18 smithi153 ceph-mon[24571]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-09-20T22:36:19.386 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:19 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/203424562' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/3265629470"}]': finished 2024-09-20T22:36:19.387 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:19 smithi153 ceph-mon[24571]: osdmap e87: 8 total, 8 up, 8 in 2024-09-20T22:36:19.387 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:19 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1173381631' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/4280258613"}]: dispatch 2024-09-20T22:36:19.387 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:19 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/4280258613"}]: dispatch 2024-09-20T22:36:19.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:19 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/203424562' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/3265629470"}]': finished 2024-09-20T22:36:19.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:19 smithi115 ceph-mon[23720]: osdmap e87: 8 total, 8 up, 8 in 2024-09-20T22:36:19.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:19 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1173381631' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/4280258613"}]: dispatch 2024-09-20T22:36:19.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:19 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/4280258613"}]: dispatch 2024-09-20T22:36:19.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:19 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/203424562' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/3265629470"}]': finished 2024-09-20T22:36:19.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:19 smithi115 ceph-mon[28216]: osdmap e87: 8 total, 8 up, 8 in 2024-09-20T22:36:19.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:19 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1173381631' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/4280258613"}]: dispatch 2024-09-20T22:36:19.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:19 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/4280258613"}]: dispatch 2024-09-20T22:36:19.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:36:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:36:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:36:20.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:20 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/4280258613"}]': finished 2024-09-20T22:36:20.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:20 smithi115 ceph-mon[23720]: osdmap e88: 8 total, 8 up, 8 in 2024-09-20T22:36:20.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:20 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/4001215111' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/2364546410"}]: dispatch 2024-09-20T22:36:20.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:20 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/2364546410"}]: dispatch 2024-09-20T22:36:20.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:20 smithi115 ceph-mon[23720]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:36:20.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:20 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/4280258613"}]': finished 2024-09-20T22:36:20.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:20 smithi115 ceph-mon[28216]: osdmap e88: 8 total, 8 up, 8 in 2024-09-20T22:36:20.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:20 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/4001215111' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/2364546410"}]: dispatch 2024-09-20T22:36:20.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:20 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/2364546410"}]: dispatch 2024-09-20T22:36:20.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:20 smithi115 ceph-mon[28216]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:36:20.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:20 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/4280258613"}]': finished 2024-09-20T22:36:20.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:20 smithi153 ceph-mon[24571]: osdmap e88: 8 total, 8 up, 8 in 2024-09-20T22:36:20.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:20 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/4001215111' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/2364546410"}]: dispatch 2024-09-20T22:36:20.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:20 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/2364546410"}]: dispatch 2024-09-20T22:36:20.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:20 smithi153 ceph-mon[24571]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:36:21.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:21 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/2364546410"}]': finished 2024-09-20T22:36:21.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:21 smithi153 ceph-mon[24571]: osdmap e89: 8 total, 8 up, 8 in 2024-09-20T22:36:21.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:21 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/588905139' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/2364546410"}]: dispatch 2024-09-20T22:36:21.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:21 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/2364546410"}]: dispatch 2024-09-20T22:36:21.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:36:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:36:21] "GET /metrics HTTP/1.1" 200 214943 "" "Prometheus/2.33.4" 2024-09-20T22:36:21.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:21 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/2364546410"}]': finished 2024-09-20T22:36:21.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:21 smithi115 ceph-mon[23720]: osdmap e89: 8 total, 8 up, 8 in 2024-09-20T22:36:21.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:21 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/588905139' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/2364546410"}]: dispatch 2024-09-20T22:36:21.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:21 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/2364546410"}]: dispatch 2024-09-20T22:36:21.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:21 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/2364546410"}]': finished 2024-09-20T22:36:21.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:21 smithi115 ceph-mon[28216]: osdmap e89: 8 total, 8 up, 8 in 2024-09-20T22:36:21.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:21 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/588905139' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/2364546410"}]: dispatch 2024-09-20T22:36:21.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:21 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/2364546410"}]: dispatch 2024-09-20T22:36:22.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:22 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/2364546410"}]': finished 2024-09-20T22:36:22.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:22 smithi153 ceph-mon[24571]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T22:36:22.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:22 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:22.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:22 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/787672169' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3595964724"}]: dispatch 2024-09-20T22:36:22.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:22 smithi153 ceph-mon[24571]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:22.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:22 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/2364546410"}]': finished 2024-09-20T22:36:22.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:22 smithi115 ceph-mon[23720]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T22:36:22.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:22 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:22.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:22 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/787672169' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3595964724"}]: dispatch 2024-09-20T22:36:22.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:22 smithi115 ceph-mon[23720]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:22.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:22 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/2364546410"}]': finished 2024-09-20T22:36:22.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:22 smithi115 ceph-mon[28216]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T22:36:22.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:22 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:22.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:22 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/787672169' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3595964724"}]: dispatch 2024-09-20T22:36:22.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:22 smithi115 ceph-mon[28216]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:23.509 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:23 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/787672169' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3595964724"}]': finished 2024-09-20T22:36:23.509 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:23 smithi115 ceph-mon[23720]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T22:36:23.509 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:23 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2188161950' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1642154865"}]: dispatch 2024-09-20T22:36:23.509 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:23 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1642154865"}]: dispatch 2024-09-20T22:36:23.509 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:23 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/787672169' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3595964724"}]': finished 2024-09-20T22:36:23.509 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:23 smithi115 ceph-mon[28216]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T22:36:23.510 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:23 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2188161950' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1642154865"}]: dispatch 2024-09-20T22:36:23.510 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:23 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1642154865"}]: dispatch 2024-09-20T22:36:23.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:23 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/787672169' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3595964724"}]': finished 2024-09-20T22:36:23.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:23 smithi153 ceph-mon[24571]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T22:36:23.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:23 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2188161950' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1642154865"}]: dispatch 2024-09-20T22:36:23.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:23 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1642154865"}]: dispatch 2024-09-20T22:36:23.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:36:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=error ts=2024-09-20T22:36:23.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-09-20T22:36:23.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:36:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:36:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2024-09-20T22:36:23.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:36:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:36:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-09-20T22:36:24.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:24 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1642154865"}]': finished 2024-09-20T22:36:24.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:24 smithi153 ceph-mon[24571]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T22:36:24.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:24 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/850085041' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1694194070"}]: dispatch 2024-09-20T22:36:24.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:24 smithi153 ceph-mon[24571]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:24.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:24 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1642154865"}]': finished 2024-09-20T22:36:24.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:24 smithi115 ceph-mon[23720]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T22:36:24.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:24 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/850085041' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1694194070"}]: dispatch 2024-09-20T22:36:24.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:24 smithi115 ceph-mon[23720]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:24.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:24 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1642154865"}]': finished 2024-09-20T22:36:24.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:24 smithi115 ceph-mon[28216]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T22:36:24.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:24 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/850085041' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1694194070"}]: dispatch 2024-09-20T22:36:24.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:24 smithi115 ceph-mon[28216]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:25.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:25 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/850085041' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1694194070"}]': finished 2024-09-20T22:36:25.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:25 smithi153 ceph-mon[24571]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T22:36:25.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:25 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2407025069' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/804615791"}]: dispatch 2024-09-20T22:36:25.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:25 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/804615791"}]: dispatch 2024-09-20T22:36:25.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:25 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/850085041' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1694194070"}]': finished 2024-09-20T22:36:25.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:25 smithi115 ceph-mon[23720]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T22:36:25.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:25 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2407025069' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/804615791"}]: dispatch 2024-09-20T22:36:25.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:25 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/804615791"}]: dispatch 2024-09-20T22:36:25.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:25 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/850085041' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1694194070"}]': finished 2024-09-20T22:36:25.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:25 smithi115 ceph-mon[28216]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T22:36:25.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:25 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2407025069' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/804615791"}]: dispatch 2024-09-20T22:36:25.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:25 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/804615791"}]: dispatch 2024-09-20T22:36:26.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:26 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/804615791"}]': finished 2024-09-20T22:36:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:26 smithi153 ceph-mon[24571]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T22:36:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:26 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/753513026' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4062808784"}]: dispatch 2024-09-20T22:36:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:26 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4062808784"}]: dispatch 2024-09-20T22:36:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:26 smithi153 ceph-mon[24571]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:26.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:26 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/804615791"}]': finished 2024-09-20T22:36:26.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:26 smithi115 ceph-mon[23720]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T22:36:26.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:26 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/753513026' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4062808784"}]: dispatch 2024-09-20T22:36:26.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:26 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4062808784"}]: dispatch 2024-09-20T22:36:26.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:26 smithi115 ceph-mon[23720]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:26.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:26 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/804615791"}]': finished 2024-09-20T22:36:26.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:26 smithi115 ceph-mon[28216]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T22:36:26.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:26 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/753513026' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4062808784"}]: dispatch 2024-09-20T22:36:26.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:26 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4062808784"}]: dispatch 2024-09-20T22:36:26.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:26 smithi115 ceph-mon[28216]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:27.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:27 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4062808784"}]': finished 2024-09-20T22:36:27.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:27 smithi153 ceph-mon[24571]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T22:36:27.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:27 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3208197057' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/267207316"}]: dispatch 2024-09-20T22:36:27.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:27 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4062808784"}]': finished 2024-09-20T22:36:27.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:27 smithi115 ceph-mon[23720]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T22:36:27.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:27 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3208197057' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/267207316"}]: dispatch 2024-09-20T22:36:27.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:27 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4062808784"}]': finished 2024-09-20T22:36:27.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:27 smithi115 ceph-mon[28216]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T22:36:27.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:27 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3208197057' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/267207316"}]: dispatch 2024-09-20T22:36:28.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:28 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3208197057' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/267207316"}]': finished 2024-09-20T22:36:28.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:28 smithi153 ceph-mon[24571]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T22:36:28.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:28 smithi153 ceph-mon[24571]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:28.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:28 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3208197057' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/267207316"}]': finished 2024-09-20T22:36:28.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:28 smithi115 ceph-mon[23720]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T22:36:28.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:28 smithi115 ceph-mon[23720]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:28.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:28 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3208197057' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/267207316"}]': finished 2024-09-20T22:36:28.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:28 smithi115 ceph-mon[28216]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T22:36:28.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:28 smithi115 ceph-mon[28216]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:29.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:36:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:36:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:36:31.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:30 smithi153 ceph-mon[24571]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s 2024-09-20T22:36:31.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:30 smithi115 ceph-mon[23720]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s 2024-09-20T22:36:31.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:30 smithi115 ceph-mon[28216]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s 2024-09-20T22:36:31.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:36:31 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:36:31] "GET /metrics HTTP/1.1" 200 214948 "" "Prometheus/2.33.4" 2024-09-20T22:36:33.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:32 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:33.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:32 smithi153 ceph-mon[24571]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:36:33.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:32 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:33.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:32 smithi115 ceph-mon[23720]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:36:33.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:32 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:33.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:32 smithi115 ceph-mon[28216]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:36:33.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:36:33 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=error ts=2024-09-20T22:36:33.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2024-09-20T22:36:35.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:34 smithi153 ceph-mon[24571]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:36:35.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:34 smithi115 ceph-mon[23720]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:36:35.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:34 smithi115 ceph-mon[28216]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:36:37.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:36 smithi153 ceph-mon[24571]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:36:37.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:36 smithi115 ceph-mon[23720]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:36:37.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:36 smithi115 ceph-mon[28216]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:36:39.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:38 smithi153 ceph-mon[24571]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-09-20T22:36:39.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:38 smithi115 ceph-mon[23720]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-09-20T22:36:39.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:38 smithi115 ceph-mon[28216]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-09-20T22:36:39.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:36:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:36:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:36:41.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:40 smithi153 ceph-mon[24571]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:41.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:40 smithi115 ceph-mon[23720]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:41.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:40 smithi115 ceph-mon[28216]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:41.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:36:41 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:36:41] "GET /metrics HTTP/1.1" 200 214948 "" "Prometheus/2.33.4" 2024-09-20T22:36:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:42 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:42 smithi153 ceph-mon[24571]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:43.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:42 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:43.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:42 smithi115 ceph-mon[23720]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:43.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:42 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:43.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:42 smithi115 ceph-mon[28216]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:45.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:44 smithi153 ceph-mon[24571]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:44 smithi115 ceph-mon[23720]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:45.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:44 smithi115 ceph-mon[28216]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:47.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:46 smithi153 ceph-mon[24571]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:47.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:46 smithi115 ceph-mon[23720]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:47.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:46 smithi115 ceph-mon[28216]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:49.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:48 smithi153 ceph-mon[24571]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:48 smithi115 ceph-mon[23720]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:49.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:48 smithi115 ceph-mon[28216]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:49.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:36:49 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:36:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:36:51.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:50 smithi153 ceph-mon[24571]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:50 smithi115 ceph-mon[23720]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:51.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:50 smithi115 ceph-mon[28216]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:51.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:36:51 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:36:51] "GET /metrics HTTP/1.1" 200 214889 "" "Prometheus/2.33.4" 2024-09-20T22:36:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:52 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:52 smithi153 ceph-mon[24571]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:52 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:52 smithi115 ceph-mon[23720]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:53.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:52 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:36:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:52 smithi115 ceph-mon[28216]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:55.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:54 smithi153 ceph-mon[24571]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:54 smithi115 ceph-mon[23720]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:55.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:54 smithi115 ceph-mon[28216]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:57.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:56 smithi153 ceph-mon[24571]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:57.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:56 smithi115 ceph-mon[23720]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:57.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:56 smithi115 ceph-mon[28216]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:36:59.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:58 smithi153 ceph-mon[24571]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:59.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:58 smithi115 ceph-mon[23720]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:59.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:58 smithi115 ceph-mon[28216]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:36:59.764 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:36:59 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:36:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:37:00.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:59 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:37:00.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:36:59 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:37:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:59 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:37:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:36:59 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:37:00.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:59 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:37:00.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:36:59 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:37:01.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:00 smithi153 ceph-mon[24571]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:00 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:37:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:00 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:37:01.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:00 smithi115 ceph-mon[23720]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:01.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:00 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:37:01.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:00 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:37:01.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:00 smithi115 ceph-mon[28216]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:01.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:00 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:37:01.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:00 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:37:01.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:37:01 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:37:01] "GET /metrics HTTP/1.1" 200 214851 "" "Prometheus/2.33.4" 2024-09-20T22:37:03.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:02 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:03.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:02 smithi153 ceph-mon[24571]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:02 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:02 smithi115 ceph-mon[23720]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:03.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:02 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:03.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:02 smithi115 ceph-mon[28216]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:05.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:04 smithi153 ceph-mon[24571]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:05.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:04 smithi115 ceph-mon[23720]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:05.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:04 smithi115 ceph-mon[28216]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:07.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:06 smithi153 ceph-mon[24571]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:07.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:06 smithi115 ceph-mon[23720]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:07.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:06 smithi115 ceph-mon[28216]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:09.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:08 smithi153 ceph-mon[24571]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:09.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:08 smithi115 ceph-mon[23720]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:09.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:08 smithi115 ceph-mon[28216]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:09.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:37:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:37:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:37:11.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:10 smithi153 ceph-mon[24571]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:10 smithi115 ceph-mon[23720]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:11.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:10 smithi115 ceph-mon[28216]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:11.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:37:11 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:37:11] "GET /metrics HTTP/1.1" 200 214851 "" "Prometheus/2.33.4" 2024-09-20T22:37:13.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:12 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:12 smithi153 ceph-mon[24571]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:12 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:12 smithi115 ceph-mon[23720]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:13.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:12 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:13.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:12 smithi115 ceph-mon[28216]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:15.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:14 smithi153 ceph-mon[24571]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:15.090 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:14 smithi115 ceph-mon[23720]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:15.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:14 smithi115 ceph-mon[28216]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:16.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:15 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:16.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:15 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:37:16.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:15 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:37:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:15 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:15 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:37:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:15 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:37:16.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:15 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:16.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:15 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:37:16.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:15 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:37:16.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:16 smithi153 ceph-mon[24571]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:16 smithi115 ceph-mon[23720]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:16 smithi115 ceph-mon[28216]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:19.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:18 smithi153 ceph-mon[24571]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:19.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:18 smithi115 ceph-mon[23720]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:19.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:18 smithi115 ceph-mon[28216]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:19.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:37:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:37:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:37:21.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:20 smithi153 ceph-mon[24571]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:21.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:20 smithi115 ceph-mon[23720]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:21.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:20 smithi115 ceph-mon[28216]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:21.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:37:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:37:21] "GET /metrics HTTP/1.1" 200 214852 "" "Prometheus/2.33.4" 2024-09-20T22:37:23.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:22 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:22 smithi153 ceph-mon[24571]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:23.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:22 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:23.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:22 smithi115 ceph-mon[23720]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:23.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:22 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:23.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:22 smithi115 ceph-mon[28216]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:25.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:24 smithi153 ceph-mon[24571]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:24 smithi115 ceph-mon[23720]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:25.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:24 smithi115 ceph-mon[28216]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:27.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:26 smithi153 ceph-mon[24571]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:26 smithi115 ceph-mon[23720]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:27.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:26 smithi115 ceph-mon[28216]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:29.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:28 smithi153 ceph-mon[24571]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:28 smithi115 ceph-mon[23720]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:29.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:28 smithi115 ceph-mon[28216]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:29.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:37:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:37:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:37:31.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:30 smithi153 ceph-mon[24571]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:31.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:30 smithi115 ceph-mon[23720]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:31.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:30 smithi115 ceph-mon[28216]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:31.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:37:31 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:37:31] "GET /metrics HTTP/1.1" 200 214854 "" "Prometheus/2.33.4" 2024-09-20T22:37:33.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:32 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:33.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:32 smithi153 ceph-mon[24571]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:33.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:32 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:33.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:32 smithi115 ceph-mon[23720]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:33.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:32 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:33.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:32 smithi115 ceph-mon[28216]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:34 smithi153 ceph-mon[24571]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:35.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:34 smithi115 ceph-mon[23720]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:35.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:34 smithi115 ceph-mon[28216]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:37.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:36 smithi153 ceph-mon[24571]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:37.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:36 smithi115 ceph-mon[23720]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:37.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:36 smithi115 ceph-mon[28216]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:39.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:38 smithi153 ceph-mon[24571]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:39.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:38 smithi115 ceph-mon[23720]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:39.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:38 smithi115 ceph-mon[28216]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:39.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:37:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:37:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:37:41.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:40 smithi153 ceph-mon[24571]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:41.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:40 smithi115 ceph-mon[23720]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:41.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:40 smithi115 ceph-mon[28216]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:41.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:37:41 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:37:41] "GET /metrics HTTP/1.1" 200 214854 "" "Prometheus/2.33.4" 2024-09-20T22:37:43.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:42 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:43.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:42 smithi115 ceph-mon[23720]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:43.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:42 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:43.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:42 smithi115 ceph-mon[28216]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:43.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:42 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:43.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:42 smithi153 ceph-mon[24571]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:45.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:44 smithi153 ceph-mon[24571]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:44 smithi115 ceph-mon[23720]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:45.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:44 smithi115 ceph-mon[28216]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:47.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:46 smithi153 ceph-mon[24571]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:47.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:46 smithi115 ceph-mon[23720]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:47.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:46 smithi115 ceph-mon[28216]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:49.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:48 smithi153 ceph-mon[24571]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:48 smithi115 ceph-mon[23720]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:49.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:48 smithi115 ceph-mon[28216]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:49.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:37:49 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:37:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:37:51.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:50 smithi115 ceph-mon[28216]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:50 smithi115 ceph-mon[23720]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:51.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:50 smithi153 ceph-mon[24571]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:51.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:37:51 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:37:51] "GET /metrics HTTP/1.1" 200 214858 "" "Prometheus/2.33.4" 2024-09-20T22:37:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:52 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:52 smithi115 ceph-mon[23720]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:53.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:52 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:52 smithi115 ceph-mon[28216]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:53.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:52 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:37:53.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:52 smithi153 ceph-mon[24571]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:55.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:54 smithi153 ceph-mon[24571]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:54 smithi115 ceph-mon[23720]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:55.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:54 smithi115 ceph-mon[28216]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:57.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:56 smithi153 ceph-mon[24571]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:57.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:56 smithi115 ceph-mon[23720]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:57.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:56 smithi115 ceph-mon[28216]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:37:59.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:58 smithi115 ceph-mon[23720]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:59.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:58 smithi115 ceph-mon[28216]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:59.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:58 smithi153 ceph-mon[24571]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:37:59.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:37:59 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:37:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:38:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:59 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:38:00.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:59 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:38:00.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:59 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:38:00.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:37:59 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:38:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:59 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:38:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:59 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:38:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:59 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:38:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:37:59 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:38:00.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:59 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:38:00.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:59 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:38:00.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:59 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:38:00.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:37:59 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:38:01.126 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:00 smithi115 ceph-mon[28216]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:01.126 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:00 smithi115 ceph-mon[23720]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:01.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:00 smithi153 ceph-mon[24571]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:01.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:38:01 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:38:01] "GET /metrics HTTP/1.1" 200 214868 "" "Prometheus/2.33.4" 2024-09-20T22:38:01.774 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-20T22:38:03.026 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:02 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:03.026 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:02 smithi115 ceph-mon[23720]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:03.026 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:02 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:03.026 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:02 smithi115 ceph-mon[28216]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:03.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:02 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:03.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:02 smithi153 ceph-mon[24571]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:04.223 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-20T22:38:04.735 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:04 smithi115 ceph-mon[23720]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:04.735 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:04 smithi115 ceph-mon[28216]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:05.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:04 smithi153 ceph-mon[24571]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:06.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:06 smithi115 ceph-mon[23720]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:06.796 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:06 smithi115 ceph-mon[28216]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:06.819 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-20T22:38:07.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:06 smithi153 ceph-mon[24571]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:09.075 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:08 smithi115 ceph-mon[23720]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:09.075 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:08 smithi115 ceph-mon[28216]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:09.098 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-20T22:38:09.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:08 smithi153 ceph-mon[24571]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:09.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:38:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:38:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:38:10.997 INFO:teuthology.orchestra.run.smithi115.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:38:10.998 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:10 smithi115 ceph-mon[23720]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:10.998 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:10 smithi115 ceph-mon[28216]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:11.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:10 smithi153 ceph-mon[24571]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:11.342 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:38:11] "GET /metrics HTTP/1.1" 200 214868 "" "Prometheus/2.33.4" 2024-09-20T22:38:11.798 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T22:38:11.807 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi115.front.sepia.ceph.com 2024-09-20T22:38:11.807 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-09-20T22:38:12.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[23720]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:12.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[23720]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:38:12.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:38:12.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:12.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:12.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:38:12.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[23720]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:38:12.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:12.152 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[23720]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:12.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[28216]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:12.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[28216]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:38:12.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:38:12.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:12.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:12.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:38:12.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[28216]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:38:12.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:12.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:11 smithi115 ceph-mon[28216]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:12.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:11 smithi153 ceph-mon[24571]: from='client.15033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:11 smithi153 ceph-mon[24571]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:38:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:11 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:38:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:11 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:11 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:11 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:38:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:11 smithi153 ceph-mon[24571]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:38:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:11 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:11 smithi153 ceph-mon[24571]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:13.923 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:38:14.706 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:38:14.706 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2m) 2m ago 3m 15.4M - ba2b418f427c cd90350c7e9a 2024-09-20T22:38:14.706 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (2m) 2m ago 2m 46.7M - 8.3.5 dad864ee21e9 1549ac2de17e 2024-09-20T22:38:14.706 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (2m) 2m ago 2m 61.4M - 3.5 e1d6a67b021e e3ff3d2789f3 2024-09-20T22:38:14.706 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443 running (7m) 2m ago 7m 410M - 17.2.0 e1d6a67b021e 983f38fdc5e6 2024-09-20T22:38:14.706 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:9283 running (9m) 2m ago 9m 455M - 17.2.0 e1d6a67b021e 8bc4016f6056 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (9m) 2m ago 9m 46.2M 2048M 17.2.0 e1d6a67b021e 0b3311e89569 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (7m) 2m ago 7m 39.5M 2048M 17.2.0 e1d6a67b021e 62b3afaddd26 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (8m) 2m ago 8m 42.7M 2048M 17.2.0 e1d6a67b021e 3ee70f11001d 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3m) 2m ago 3m 13.7M - 1dbe0e931976 111c7bbee4de 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (3m) 2m ago 3m 14.0M - 1dbe0e931976 5717e9268a3a 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (7m) 2m ago 7m 49.7M 4016M 17.2.0 e1d6a67b021e 06f9f63b0da2 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (7m) 2m ago 6m 49.5M 4016M 17.2.0 e1d6a67b021e b68eba671ad6 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (6m) 2m ago 6m 45.3M 4016M 17.2.0 e1d6a67b021e a368d2632c3f 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6m) 2m ago 6m 49.0M 4016M 17.2.0 e1d6a67b021e d4ef32545846 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (5m) 2m ago 5m 50.5M 4272M 17.2.0 e1d6a67b021e e9316b3d7317 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (5m) 2m ago 5m 45.5M 4272M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (5m) 2m ago 5m 44.2M 4272M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (4m) 2m ago 4m 47.9M 4272M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:38:14.707 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (2m) 2m ago 3m 42.7M - 514e6a882f6e 7dda2e0d2690 2024-09-20T22:38:14.708 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (2m) 2m ago 2m 71.2M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:38:14.708 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (2m) 2m ago 2m 70.9M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:38:14.708 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (2m) 2m ago 2m 70.3M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:38:14.708 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (2m) 2m ago 2m 67.4M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:38:15.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:14 smithi153 ceph-mon[24571]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:15.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:14 smithi153 ceph-mon[24571]: from='client.24943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:15.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:14 smithi115 ceph-mon[23720]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:15.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:14 smithi115 ceph-mon[23720]: from='client.24943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:15.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:14 smithi115 ceph-mon[28216]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:15.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:14 smithi115 ceph-mon[28216]: from='client.24943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:15.256 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:38:15.256 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:38:15.256 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-09-20T22:38:15.256 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:38:15.256 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:38:15.256 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T22:38:15.256 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:38:15.257 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:38:15.257 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T22:38:15.257 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:38:15.257 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": {}, 2024-09-20T22:38:15.257 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:38:15.257 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:38:15.257 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:38:15.257 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:38:15.257 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-09-20T22:38:15.257 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:38:15.257 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:38:15.615 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:38:15.615 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:38:15.615 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:38:15.615 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [], 2024-09-20T22:38:15.615 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "", 2024-09-20T22:38:15.615 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 image" 2024-09-20T22:38:15.615 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:38:15.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:15 smithi115 ceph-mon[23720]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:15.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:15 smithi115 ceph-mon[23720]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:15.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:15 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/4089060392' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:38:15.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:15 smithi115 ceph-mon[28216]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:15.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:15 smithi115 ceph-mon[28216]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:15.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:15 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/4089060392' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:38:16.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:15 smithi153 ceph-mon[24571]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:16.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:15 smithi153 ceph-mon[24571]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:16.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:15 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/4089060392' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:38:16.036 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:38:17.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:16 smithi153 ceph-mon[24571]: from='client.24961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:17.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:16 smithi153 ceph-mon[24571]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:17.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:16 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2222794854' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:38:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:16 smithi115 ceph-mon[23720]: from='client.24961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:16 smithi115 ceph-mon[23720]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:16 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2222794854' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:38:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:16 smithi115 ceph-mon[28216]: from='client.24961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:16 smithi115 ceph-mon[28216]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:16 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2222794854' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:38:19.139 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:18 smithi115 ceph-mon[23720]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:19.140 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:18 smithi115 ceph-mon[28216]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:19.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:18 smithi153 ceph-mon[24571]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:19.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:38:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:38:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:38:21.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:20 smithi115 ceph-mon[23720]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:21.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:20 smithi115 ceph-mon[28216]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:21.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:20 smithi153 ceph-mon[24571]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:21.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:38:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:38:21] "GET /metrics HTTP/1.1" 200 214851 "" "Prometheus/2.33.4" 2024-09-20T22:38:23.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:22 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:23.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:22 smithi115 ceph-mon[23720]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:23.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:22 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:23.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:22 smithi115 ceph-mon[28216]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:23.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:22 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:23.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:22 smithi153 ceph-mon[24571]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:25.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:24 smithi153 ceph-mon[24571]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:24 smithi115 ceph-mon[23720]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:25.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:24 smithi115 ceph-mon[28216]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:26 smithi115 ceph-mon[23720]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:27.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:26 smithi115 ceph-mon[28216]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:27.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:26 smithi153 ceph-mon[24571]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:28 smithi115 ceph-mon[23720]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:29.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:28 smithi115 ceph-mon[28216]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:29.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:28 smithi153 ceph-mon[24571]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:29.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:38:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:38:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:38:30.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:30 smithi115 ceph-mon[23720]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:30.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:30 smithi115 ceph-mon[28216]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:31.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:30 smithi153 ceph-mon[24571]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:31.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:38:31 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:38:31] "GET /metrics HTTP/1.1" 200 214823 "" "Prometheus/2.33.4" 2024-09-20T22:38:33.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:32 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:33.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:32 smithi153 ceph-mon[24571]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:33.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:32 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:33.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:32 smithi115 ceph-mon[23720]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:33.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:32 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:33.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:32 smithi115 ceph-mon[28216]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:35.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:34 smithi153 ceph-mon[24571]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:35.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:34 smithi115 ceph-mon[23720]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:35.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:34 smithi115 ceph-mon[28216]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:36.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:36 smithi115 ceph-mon[23720]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:36.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:36 smithi115 ceph-mon[28216]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:37.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:36 smithi153 ceph-mon[24571]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:38.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:38 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:38:38.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:38 smithi153 ceph-mon[24571]: Upgrade: Target is version 19.1.1-367-g9e2ac70b (unknown) 2024-09-20T22:38:38.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:38 smithi153 ceph-mon[24571]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, digests ['quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5'] 2024-09-20T22:38:38.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:38 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:38.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:38 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:38:38.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:38 smithi153 ceph-mon[24571]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T22:38:38.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:38 smithi153 ceph-mon[24571]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 on smithi153 2024-09-20T22:38:38.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:38 smithi153 ceph-mon[24571]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:38.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:38:38.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[23720]: Upgrade: Target is version 19.1.1-367-g9e2ac70b (unknown) 2024-09-20T22:38:38.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[23720]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, digests ['quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5'] 2024-09-20T22:38:38.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:38.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:38:38.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[23720]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T22:38:38.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[23720]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 on smithi153 2024-09-20T22:38:38.806 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[23720]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:38.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:38:38.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[28216]: Upgrade: Target is version 19.1.1-367-g9e2ac70b (unknown) 2024-09-20T22:38:38.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[28216]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, digests ['quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5'] 2024-09-20T22:38:38.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:38.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:38:38.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[28216]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T22:38:38.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[28216]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 on smithi153 2024-09-20T22:38:38.807 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:38 smithi115 ceph-mon[28216]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:39.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:38:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:38:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:38:41.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:40 smithi115 ceph-mon[23720]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:41.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:40 smithi115 ceph-mon[28216]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:41.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:40 smithi153 ceph-mon[24571]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:41.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:38:41 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:38:41] "GET /metrics HTTP/1.1" 200 214823 "" "Prometheus/2.33.4" 2024-09-20T22:38:43.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:42 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:43.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:42 smithi115 ceph-mon[23720]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:43.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:42 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:43.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:42 smithi115 ceph-mon[28216]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:43.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:42 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:43.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:42 smithi153 ceph-mon[24571]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:45.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:44 smithi153 ceph-mon[24571]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:44 smithi115 ceph-mon[23720]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:45.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:44 smithi115 ceph-mon[28216]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:46.358 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:38:46.948 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:38:46.948 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3m) 2m ago 3m 15.4M - ba2b418f427c cd90350c7e9a 2024-09-20T22:38:46.948 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (3m) 2m ago 3m 46.7M - 8.3.5 dad864ee21e9 1549ac2de17e 2024-09-20T22:38:46.948 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (2m) 2m ago 2m 61.4M - 3.5 e1d6a67b021e e3ff3d2789f3 2024-09-20T22:38:46.948 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443 running (8m) 2m ago 8m 410M - 17.2.0 e1d6a67b021e 983f38fdc5e6 2024-09-20T22:38:46.948 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:9283 running (9m) 2m ago 9m 455M - 17.2.0 e1d6a67b021e 8bc4016f6056 2024-09-20T22:38:46.948 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (9m) 2m ago 9m 46.2M 2048M 17.2.0 e1d6a67b021e 0b3311e89569 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (8m) 2m ago 8m 39.5M 2048M 17.2.0 e1d6a67b021e 62b3afaddd26 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (8m) 2m ago 8m 42.7M 2048M 17.2.0 e1d6a67b021e 3ee70f11001d 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4m) 2m ago 4m 13.7M - 1dbe0e931976 111c7bbee4de 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (4m) 2m ago 4m 14.0M - 1dbe0e931976 5717e9268a3a 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (7m) 2m ago 7m 49.7M 4016M 17.2.0 e1d6a67b021e 06f9f63b0da2 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (7m) 2m ago 7m 49.5M 4016M 17.2.0 e1d6a67b021e b68eba671ad6 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (7m) 2m ago 7m 45.3M 4016M 17.2.0 e1d6a67b021e a368d2632c3f 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (6m) 2m ago 6m 49.0M 4016M 17.2.0 e1d6a67b021e d4ef32545846 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (6m) 2m ago 6m 50.5M 4272M 17.2.0 e1d6a67b021e e9316b3d7317 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (5m) 2m ago 5m 45.5M 4272M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (5m) 2m ago 5m 44.2M 4272M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (5m) 2m ago 5m 47.9M 4272M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (3m) 2m ago 4m 42.7M - 514e6a882f6e 7dda2e0d2690 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (2m) 2m ago 2m 71.2M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:38:46.949 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (2m) 2m ago 2m 70.9M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:38:46.950 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (2m) 2m ago 2m 70.3M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:38:46.950 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (2m) 2m ago 2m 67.4M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:38:47.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:46 smithi115 ceph-mon[23720]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:47.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:46 smithi115 ceph-mon[28216]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:47.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:46 smithi153 ceph-mon[24571]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:47.301 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:38:47.302 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:38:47.302 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-09-20T22:38:47.302 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:38:47.302 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:38:47.302 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T22:38:47.302 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:38:47.302 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:38:47.302 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T22:38:47.302 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:38:47.302 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": {}, 2024-09-20T22:38:47.302 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:38:47.302 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:38:47.303 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:38:47.303 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:38:47.303 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-09-20T22:38:47.303 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:38:47.303 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:38:47.599 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:38:47.599 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:38:47.599 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:38:47.599 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [], 2024-09-20T22:38:47.599 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "0/23 daemons upgraded", 2024-09-20T22:38:47.599 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 image on host smithi153" 2024-09-20T22:38:47.599 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:38:47.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:47 smithi115 ceph-mon[23720]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:47.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:47 smithi115 ceph-mon[23720]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:47.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:47 smithi115 ceph-mon[23720]: from='client.24985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:47.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:47 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1507412169' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:38:47.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:47 smithi115 ceph-mon[28216]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:47.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:47 smithi115 ceph-mon[28216]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:47.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:47 smithi115 ceph-mon[28216]: from='client.24985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:47.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:47 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1507412169' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:38:47.952 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:38:48.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:47 smithi153 ceph-mon[24571]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:48.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:47 smithi153 ceph-mon[24571]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:48.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:47 smithi153 ceph-mon[24571]: from='client.24985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:48.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:47 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1507412169' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:38:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:48 smithi115 ceph-mon[23720]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:48 smithi115 ceph-mon[23720]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:48 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3612987517' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:38:49.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:48 smithi115 ceph-mon[28216]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:49.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:48 smithi115 ceph-mon[28216]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:49.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:48 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3612987517' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:38:49.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:48 smithi153 ceph-mon[24571]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:49.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:48 smithi153 ceph-mon[24571]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:49.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:48 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3612987517' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:38:49.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:38:49 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:38:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:38:51.064 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:50 smithi115 ceph-mon[23720]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:51.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:50 smithi115 ceph-mon[28216]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:51.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:50 smithi153 ceph-mon[24571]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:51.373 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:38:51 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:38:51] "GET /metrics HTTP/1.1" 200 214799 "" "Prometheus/2.33.4" 2024-09-20T22:38:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:52 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:52 smithi115 ceph-mon[23720]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:53.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:52 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:52 smithi115 ceph-mon[28216]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:53.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:52 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:38:53.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:52 smithi153 ceph-mon[24571]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:53.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:38:53 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:38:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-09-20T22:38:53.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:38:53 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:38:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2024-09-20T22:38:55.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:54 smithi153 ceph-mon[24571]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:54 smithi115 ceph-mon[23720]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:55.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:54 smithi115 ceph-mon[28216]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:57.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:56 smithi115 ceph-mon[23720]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:57.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:56 smithi115 ceph-mon[28216]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:57.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:56 smithi153 ceph-mon[24571]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:38:59.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:58 smithi115 ceph-mon[23720]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:59.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:58 smithi115 ceph-mon[28216]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:59.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:58 smithi153 ceph-mon[24571]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:38:59.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:38:59 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[25748]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:38:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:39:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:59 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:39:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:59 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:39:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:59 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:39:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:38:59 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:39:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:59 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:39:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:59 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:39:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:59 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:39:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:38:59 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:39:00.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:59 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:39:00.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:59 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:39:00.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:59 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:39:00.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:38:59 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:39:01.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:00 smithi115 ceph-mon[23720]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:01.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:00 smithi115 ceph-mon[28216]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:01.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:00 smithi153 ceph-mon[24571]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:01.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:01 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:39:01] "GET /metrics HTTP/1.1" 200 214799 "" "Prometheus/2.33.4" 2024-09-20T22:39:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:02 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:02 smithi115 ceph-mon[23720]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:02 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:02 smithi115 ceph-mon[28216]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:03.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:02 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:03.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:02 smithi153 ceph-mon[24571]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:03.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:03 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=error ts=2024-09-20T22:39:03.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2024-09-20T22:39:03.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:03 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2024-09-20T22:39:03.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:03 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-09-20T22:39:05.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:04 smithi153 ceph-mon[24571]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:05.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:04 smithi115 ceph-mon[23720]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:05.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:04 smithi115 ceph-mon[28216]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:07.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:06 smithi153 ceph-mon[24571]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:07.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:06 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:07.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:06 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:39:07.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:06 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:39:07.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:06 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:39:07.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:06 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:07.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:06 smithi115 ceph-mon[28216]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:07.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:06 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:07.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:06 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:39:07.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:06 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:39:07.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:06 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:39:07.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:06 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:07.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:06 smithi115 ceph-mon[23720]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:07.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:06 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:07.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:06 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:39:07.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:06 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:39:07.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:06 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:39:07.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:06 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:07.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:07 smithi153 ceph-mon[24571]: Upgrade: Updating mgr.x 2024-09-20T22:39:07.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:07 smithi153 ceph-mon[24571]: Deploying daemon mgr.x on smithi153 2024-09-20T22:39:08.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:07 smithi115 ceph-mon[23720]: Upgrade: Updating mgr.x 2024-09-20T22:39:08.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:07 smithi115 ceph-mon[23720]: Deploying daemon mgr.x on smithi153 2024-09-20T22:39:08.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:07 smithi115 ceph-mon[28216]: Upgrade: Updating mgr.x 2024-09-20T22:39:08.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:07 smithi115 ceph-mon[28216]: Deploying daemon mgr.x on smithi153 2024-09-20T22:39:08.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:08 smithi153 systemd[1]: Stopping Ceph mgr.x for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:39:08.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:08 smithi153 bash[45481]: Error: no container with name or ID "ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr.x" found: no such container 2024-09-20T22:39:08.649 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:08 smithi153 podman[45491]: 2024-09-20 22:39:08.385786053 +0000 UTC m=+0.156067914 container died 983f38fdc5e61f849c84e71a8e7dbee4c2f0fd81a363afc32ff5b21f3f75c9c9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, version=8, RELEASE=HEAD, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, architecture=x86_64) 2024-09-20T22:39:09.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:08 smithi153 ceph-mon[24571]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:09.008 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:08 smithi153 podman[45491]: 2024-09-20 22:39:08.737916916 +0000 UTC m=+0.508198772 container cleanup 983f38fdc5e61f849c84e71a8e7dbee4c2f0fd81a363afc32ff5b21f3f75c9c9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, release=754, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, name=centos-stream, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , RELEASE=HEAD, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8) 2024-09-20T22:39:09.008 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:08 smithi153 bash[45491]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x 2024-09-20T22:39:09.008 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:08 smithi153 bash[45518]: Error: no container with name or ID "ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr.x" found: no such container 2024-09-20T22:39:09.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:08 smithi115 ceph-mon[23720]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:09.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:08 smithi115 ceph-mon[28216]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:09.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 podman[45502]: 2024-09-20 22:39:09.006032836 +0000 UTC m=+0.623072228 container remove 983f38fdc5e61f849c84e71a8e7dbee4c2f0fd81a363afc32ff5b21f3f75c9c9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, name=centos-stream, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, architecture=x86_64, distribution-scope=public, vendor=Red Hat, Inc.) 2024-09-20T22:39:09.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-20T22:39:09.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-20T22:39:09.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 systemd[1]: Stopped Ceph mgr.x for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:39:09.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.x.service: Consumed 16.767s CPU time. 2024-09-20T22:39:09.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 systemd[1]: Starting Ceph mgr.x for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:39:09.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:09 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:09.350Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-09-20T22:39:09.705 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 podman[45617]: 2024-09-20 22:39:09.336516551 +0000 UTC m=+0.015901238 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:39:09.705 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 podman[45617]: 2024-09-20 22:39:09.440773137 +0000 UTC m=+0.120157877 container create e181bec15dad02e43f92269f1dfd2556c649c8ac9685f57d168714f8546c7865 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-20T22:39:09.706 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 podman[45617]: 2024-09-20 22:39:09.676425631 +0000 UTC m=+0.355810309 container init e181bec15dad02e43f92269f1dfd2556c649c8ac9685f57d168714f8546c7865 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:39:09.706 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 podman[45617]: 2024-09-20 22:39:09.679266722 +0000 UTC m=+0.358651411 container start e181bec15dad02e43f92269f1dfd2556c649c8ac9685f57d168714f8546c7865 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-20T22:39:10.028 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 ceph-mgr[45636]: -- 172.21.15.153:0/2131267551 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56402040b0e0 con 0x5640203e9000 2024-09-20T22:39:10.028 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 bash[45617]: e181bec15dad02e43f92269f1dfd2556c649c8ac9685f57d168714f8546c7865 2024-09-20T22:39:10.028 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 systemd[1]: Started Ceph mgr.x for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:39:10.028 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:09.805+0000 7faede144040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:39:10.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:09.867+0000 7faede144040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:39:10.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:10 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:10.026+0000 7faede144040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:39:10.278 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:10 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:10.107+0000 7faede144040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:39:10.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:10 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:10.266+0000 7faede144040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:39:10.742 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:10 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:10.412+0000 7faede144040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:39:10.742 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:10 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:10.740+0000 7faede144040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:39:11.001 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:10 smithi153 ceph-mon[24571]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:11.001 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:10 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:11.001 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:10 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:11.001 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:10 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:11.001 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:10 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:10.809+0000 7faede144040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:39:11.001 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:10 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:10.998+0000 7faede144040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:39:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:10 smithi115 ceph-mon[23720]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:10 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:10 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:10 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:11.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:10 smithi115 ceph-mon[28216]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:11.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:10 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:11.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:10 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:11.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:10 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:11.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:11 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:39:11] "GET /metrics HTTP/1.1" 200 214799 "" "Prometheus/2.33.4" 2024-09-20T22:39:12.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:11 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:11.952+0000 7faede144040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:39:12.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:12 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:12.025+0000 7faede144040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:39:12.918 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:12 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:12.918 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:12 smithi153 ceph-mon[24571]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:12.918 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:12 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:12.811+0000 7faede144040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:39:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:12 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:12 smithi115 ceph-mon[23720]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:13.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:12 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:12 smithi115 ceph-mon[28216]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:13.278 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:12 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:12.934+0000 7faede144040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:39:13.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:13.084+0000 7faede144040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:39:13.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:13.152+0000 7faede144040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:39:13.587 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:13.309+0000 7faede144040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T22:39:13.588 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:13.378+0000 7faede144040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:39:13.588 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:13.441+0000 7faede144040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:39:13.893 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:13.611+0000 7faede144040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:39:13.893 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:13.676+0000 7faede144040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:39:13.893 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:13.745+0000 7faede144040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:39:13.894 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:13.830+0000 7faede144040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:39:13.894 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:13.891+0000 7faede144040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:39:13.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=error ts=2024-09-20T22:39:13.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2024-09-20T22:39:13.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:13.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-09-20T22:39:13.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:13.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2024-09-20T22:39:14.278 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:13.957+0000 7faede144040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:39:14.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:14 smithi153 ceph-mon[24571]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:14.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:14 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:14.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:14 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:14.833 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:14.825+0000 7faede144040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:39:14.833 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:39:14] ENGINE Bus STARTING 2024-09-20T22:39:15.064 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:14 smithi115 ceph-mon[23720]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:15.064 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:14 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:15.064 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:14 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:15.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:14 smithi115 ceph-mon[28216]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:15.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:14 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:15.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:14 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:15.085 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: CherryPy Checker: 2024-09-20T22:39:15.085 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: The Application mounted at '' has an empty config. 2024-09-20T22:39:15.085 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:15.085 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:39:14] ENGINE Serving on http://:::9283 2024-09-20T22:39:15.085 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:39:14] ENGINE Bus STARTED 2024-09-20T22:39:15.404 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:15.063Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-09-20T22:39:15.979 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:15 smithi153 ceph-mon[24571]: Standby manager daemon x restarted 2024-09-20T22:39:15.979 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:15 smithi153 ceph-mon[24571]: Standby manager daemon x started 2024-09-20T22:39:15.979 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:15 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:39:15.979 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:15 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:39:15.979 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:15 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:39:15.979 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:15 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:39:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:15 smithi115 ceph-mon[23720]: Standby manager daemon x restarted 2024-09-20T22:39:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:15 smithi115 ceph-mon[23720]: Standby manager daemon x started 2024-09-20T22:39:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:15 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:39:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:15 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:39:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:15 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:39:16.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:15 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:39:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:15 smithi115 ceph-mon[28216]: Standby manager daemon x restarted 2024-09-20T22:39:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:15 smithi115 ceph-mon[28216]: Standby manager daemon x started 2024-09-20T22:39:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:15 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:39:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:15 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:39:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:15 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:39:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:15 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:39:17.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:16 smithi153 ceph-mon[24571]: mgrmap e21: y(active, since 5m), standbys: x 2024-09-20T22:39:17.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:16 smithi153 ceph-mon[24571]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:16 smithi115 ceph-mon[23720]: mgrmap e21: y(active, since 5m), standbys: x 2024-09-20T22:39:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:16 smithi115 ceph-mon[23720]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:16 smithi115 ceph-mon[28216]: mgrmap e21: y(active, since 5m), standbys: x 2024-09-20T22:39:17.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:16 smithi115 ceph-mon[28216]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:18.278 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:39:19.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[23720]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:19.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:19.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:19.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:19.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[23720]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T22:39:19.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[23720]: Failing over to other MGR 2024-09-20T22:39:19.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[23720]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-20T22:39:19.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-20T22:39:19.096 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[23720]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T22:39:19.097 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:18.845+0000 7fef8d44e700 -1 mgr handle_mgr_map I was active but no longer am 2024-09-20T22:39:19.097 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ignoring --setuser ceph since I am not root 2024-09-20T22:39:19.097 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ignoring --setgroup ceph since I am not root 2024-09-20T22:39:19.097 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mgr[23952]: -- 172.21.15.115:0/1863466519 <== mon.1 v2:172.21.15.115:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562906c26340 con 0x562906bfec00 2024-09-20T22:39:19.097 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:19.013+0000 7fd5eabe6000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:39:19.097 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:19.094+0000 7fd5eabe6000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:39:19.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[28216]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:19.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:19.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:19.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:19.097 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[28216]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T22:39:19.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[28216]: Failing over to other MGR 2024-09-20T22:39:19.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[28216]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-20T22:39:19.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-20T22:39:19.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:18 smithi115 ceph-mon[28216]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T22:39:19.132 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:18 smithi153 ceph-mon[24571]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:19.133 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:18 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:19.133 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:18 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' 2024-09-20T22:39:19.133 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:18 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:19.133 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:18 smithi153 ceph-mon[24571]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-20T22:39:19.133 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:18 smithi153 ceph-mon[24571]: Failing over to other MGR 2024-09-20T22:39:19.133 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:18 smithi153 ceph-mon[24571]: from='mgr.14427 172.21.15.115:0/1612835689' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-20T22:39:19.133 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:18 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-20T22:39:19.133 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:18 smithi153 ceph-mon[24571]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T22:39:19.133 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:39:18] ENGINE Bus STOPPING 2024-09-20T22:39:19.133 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:39:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:39:19.133 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:39:18] ENGINE Bus STOPPED 2024-09-20T22:39:19.134 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:39:19] ENGINE Bus STARTING 2024-09-20T22:39:19.134 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:39:19] ENGINE Serving on http://:::9283 2024-09-20T22:39:19.359 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:19.267+0000 7fd5eabe6000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:39:19.386 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:39:19] ENGINE Bus STARTED 2024-09-20T22:39:19.386 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:39:19] "GET /metrics HTTP/1.1" 200 35164 "" "Prometheus/2.33.4" 2024-09-20T22:39:19.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:19.357+0000 7fd5eabe6000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:39:19.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:19.496+0000 7fd5eabe6000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:39:19.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:19.481Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.115:8443//api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused" 2024-09-20T22:39:19.880 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:39:19.880 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3m) 3m ago 4m 15.4M - ba2b418f427c cd90350c7e9a 2024-09-20T22:39:19.881 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (3m) 5s ago 3m 55.3M - 8.3.5 dad864ee21e9 1549ac2de17e 2024-09-20T22:39:19.881 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (3m) 3m ago 3m 61.4M - 3.5 e1d6a67b021e e3ff3d2789f3 2024-09-20T22:39:19.881 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283 running (10s) 5s ago 8m 65.6M - 19.1.1-367-g9e2ac70b cc9251b744a3 e181bec15dad 2024-09-20T22:39:19.881 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:9283 running (10m) 3m ago 10m 455M - 17.2.0 e1d6a67b021e 8bc4016f6056 2024-09-20T22:39:19.881 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (10m) 3m ago 10m 46.2M 2048M 17.2.0 e1d6a67b021e 0b3311e89569 2024-09-20T22:39:19.881 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (9m) 5s ago 9m 40.7M 2048M 17.2.0 e1d6a67b021e 62b3afaddd26 2024-09-20T22:39:19.881 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (9m) 3m ago 9m 42.7M 2048M 17.2.0 e1d6a67b021e 3ee70f11001d 2024-09-20T22:39:19.881 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4m) 3m ago 4m 13.7M - 1dbe0e931976 111c7bbee4de 2024-09-20T22:39:19.881 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (4m) 5s ago 4m 17.5M - 1dbe0e931976 5717e9268a3a 2024-09-20T22:39:19.881 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (8m) 3m ago 8m 49.7M 4016M 17.2.0 e1d6a67b021e 06f9f63b0da2 2024-09-20T22:39:19.881 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (8m) 3m ago 8m 49.5M 4016M 17.2.0 e1d6a67b021e b68eba671ad6 2024-09-20T22:39:19.881 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (7m) 3m ago 7m 45.3M 4016M 17.2.0 e1d6a67b021e a368d2632c3f 2024-09-20T22:39:19.881 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (7m) 3m ago 7m 49.0M 4016M 17.2.0 e1d6a67b021e d4ef32545846 2024-09-20T22:39:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (6m) 5s ago 6m 53.6M 4272M 17.2.0 e1d6a67b021e e9316b3d7317 2024-09-20T22:39:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (6m) 5s ago 6m 49.2M 4272M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:39:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (6m) 5s ago 6m 47.2M 4272M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:39:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (5m) 5s ago 5m 49.9M 4272M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:39:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (3m) 5s ago 5m 65.9M - 514e6a882f6e 7dda2e0d2690 2024-09-20T22:39:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (3m) 3m ago 3m 71.2M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:39:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (3m) 5s ago 3m 72.2M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:39:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (3m) 3m ago 3m 70.3M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:39:19.882 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (3m) 5s ago 3m 68.8M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:39:20.056 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:19.712+0000 7fd5eabe6000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:39:20.057 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-20T22:39:20.057 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: mgrmap e22: x(active, starting, since 0.992505s) 2024-09-20T22:39:20.057 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:39:20.058 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:39:20.058 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:39:20.058 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:39:20.058 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:39:20.058 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:39:20.058 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:20.059 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:20.059 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:39:20.059 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:39:20.059 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:39:20.059 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:39:20.059 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:39:20.059 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:39:20.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:39:20.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: Manager daemon x is now available 2024-09-20T22:39:20.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:20.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: Queued rgw.foo for migration 2024-09-20T22:39:20.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:20.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: Queued rgw.smpl for migration 2024-09-20T22:39:20.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-20T22:39:20.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-20T22:39:20.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:20.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-20T22:39:20.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-20T22:39:20.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-20T22:39:20.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: Checking for cert/key for grafana.a 2024-09-20T22:39:20.061 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:20.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:39:20.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:39:20.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:39:20.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T22:39:20.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T22:39:20.063 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: Deploying cephadm binary to smithi115 2024-09-20T22:39:20.063 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[23720]: Deploying cephadm binary to smithi153 2024-09-20T22:39:20.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-20T22:39:20.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: mgrmap e22: x(active, starting, since 0.992505s) 2024-09-20T22:39:20.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:39:20.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: Manager daemon x is now available 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: Queued rgw.foo for migration 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: Queued rgw.smpl for migration 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: Checking for cert/key for grafana.a 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:39:20.065 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:39:20.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T22:39:20.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T22:39:20.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: Deploying cephadm binary to smithi115 2024-09-20T22:39:20.066 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:19 smithi115 ceph-mon[28216]: Deploying cephadm binary to smithi153 2024-09-20T22:39:20.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.14427 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-20T22:39:20.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: mgrmap e22: x(active, starting, since 0.992505s) 2024-09-20T22:39:20.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:39:20.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: Manager daemon x is now available 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: Queued rgw.foo for migration 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: Queued rgw.smpl for migration 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: Checking for cert/key for grafana.a 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:39:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:39:20.282 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T22:39:20.282 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-20T22:39:20.282 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: Deploying cephadm binary to smithi115 2024-09-20T22:39:20.282 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:19 smithi153 ceph-mon[24571]: Deploying cephadm binary to smithi153 2024-09-20T22:39:20.293 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:39:20.293 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:39:20.293 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-09-20T22:39:20.293 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:39:20.293 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:39:20.293 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T22:39:20.293 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:39:20.293 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:39:20.294 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T22:39:20.294 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:39:20.294 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": {}, 2024-09-20T22:39:20.294 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:39:20.294 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:39:20.294 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:39:20.294 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:39:20.294 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-09-20T22:39:20.294 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T22:39:20.294 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:39:20.294 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:39:20.317 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:20.055+0000 7fd5eabe6000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:39:20.317 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:20.145+0000 7fd5eabe6000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:39:20.317 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:20.316+0000 7fd5eabe6000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:39:20.662 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:39:20.662 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:39:20.662 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:39:20.662 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:39:20.662 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [], 2024-09-20T22:39:20.663 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "1/23 daemons upgraded", 2024-09-20T22:39:20.663 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "", 2024-09-20T22:39:20.663 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:39:20.663 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:39:21.104 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:39:21.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[28216]: mgrmap e23: x(active, since 2s) 2024-09-20T22:39:21.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[28216]: from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:21.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[28216]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:39:21.104 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2666723492' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:39:21.105 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:21.105 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:21.105 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[28216]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:21.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[23720]: mgrmap e23: x(active, since 2s) 2024-09-20T22:39:21.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[23720]: from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:21.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[23720]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:39:21.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2666723492' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:39:21.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:21.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:21.105 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:20 smithi115 ceph-mon[23720]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:21.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:20 smithi153 ceph-mon[24571]: mgrmap e23: x(active, since 2s) 2024-09-20T22:39:21.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:20 smithi153 ceph-mon[24571]: from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:21.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:20 smithi153 ceph-mon[24571]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:39:21.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:20 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2666723492' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:39:21.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:20 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:21.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:20 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:21.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:20 smithi153 ceph-mon[24571]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:21.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:21.446+0000 7fd5eabe6000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:39:21.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:21.538+0000 7fd5eabe6000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:39:21.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:21.426+0000 7fae368b6640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:22:39:21] ENGINE Error in HTTPServer.serve 2024-09-20T22:39:21.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: Traceback (most recent call last): 2024-09-20T22:39:21.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:39:21.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: self._connections.run(self.expiration_interval) 2024-09-20T22:39:21.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:39:21.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: self._run(expiration_interval) 2024-09-20T22:39:21.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:39:21.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:39:21.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:39:21.780 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:39:21.780 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:39:21.780 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: s = self.context.wrap_socket( 2024-09-20T22:39:21.780 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:39:21.780 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: return self.sslsocket_class._create( 2024-09-20T22:39:21.780 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:39:21.780 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: self.do_handshake() 2024-09-20T22:39:21.780 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:39:21.780 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: self._sslobj.do_handshake() 2024-09-20T22:39:21.780 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:39:21.780 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[23720]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:39:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1098618103' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:39:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[23720]: [20/Sep/2024:22:39:21] ENGINE Bus STARTING 2024-09-20T22:39:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[23720]: mgrmap e24: x(active, since 3s) 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[23720]: [20/Sep/2024:22:39:21] ENGINE Serving on https://172.21.15.153:7150 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[23720]: [20/Sep/2024:22:39:21] ENGINE Error in HTTPServer.serve 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout: Traceback (most recent call last): 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout: self._run(expiration_interval) 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout: s = self.context.wrap_socket( 2024-09-20T22:39:22.155 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:39:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout: return self.sslsocket_class._create( 2024-09-20T22:39:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:39:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout: self.do_handshake() 2024-09-20T22:39:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:39:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout: self._sslobj.do_handshake() 2024-09-20T22:39:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:39:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[23720]: [20/Sep/2024:22:39:21] ENGINE Serving on http://172.21.15.153:8765 2024-09-20T22:39:22.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[23720]: [20/Sep/2024:22:39:21] ENGINE Bus STARTED 2024-09-20T22:39:22.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[28216]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1098618103' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[28216]: [20/Sep/2024:22:39:21] ENGINE Bus STARTING 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[28216]: mgrmap e24: x(active, since 3s) 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[28216]: [20/Sep/2024:22:39:21] ENGINE Serving on https://172.21.15.153:7150 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[28216]: [20/Sep/2024:22:39:21] ENGINE Error in HTTPServer.serve 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._run(expiration_interval) 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:39:22.157 INFO:journalctl@ceph.mon.c.smithi115.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:39:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:39:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout: s = self.context.wrap_socket( 2024-09-20T22:39:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:39:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.sslsocket_class._create( 2024-09-20T22:39:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:39:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.do_handshake() 2024-09-20T22:39:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:39:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._sslobj.do_handshake() 2024-09-20T22:39:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:39:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[28216]: [20/Sep/2024:22:39:21] ENGINE Serving on http://172.21.15.153:8765 2024-09-20T22:39:22.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:21 smithi115 ceph-mon[28216]: [20/Sep/2024:22:39:21] ENGINE Bus STARTED 2024-09-20T22:39:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-mon[24571]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:39:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1098618103' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:39:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-mon[24571]: [20/Sep/2024:22:39:21] ENGINE Bus STARTING 2024-09-20T22:39:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-mon[24571]: mgrmap e24: x(active, since 3s) 2024-09-20T22:39:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-mon[24571]: [20/Sep/2024:22:39:21] ENGINE Serving on https://172.21.15.153:7150 2024-09-20T22:39:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-mon[24571]: [20/Sep/2024:22:39:21] ENGINE Error in HTTPServer.serve 2024-09-20T22:39:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-09-20T22:39:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:39:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:39:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:39:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout: self._run(expiration_interval) 2024-09-20T22:39:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:39:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:39:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:39:22.280 INFO:journalctl@ceph.mon.b.smithi153.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:39:22.281 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:39:22.281 INFO:journalctl@ceph.mon.b.smithi153.stdout: s = self.context.wrap_socket( 2024-09-20T22:39:22.281 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:39:22.281 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.sslsocket_class._create( 2024-09-20T22:39:22.281 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:39:22.281 INFO:journalctl@ceph.mon.b.smithi153.stdout: self.do_handshake() 2024-09-20T22:39:22.281 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:39:22.282 INFO:journalctl@ceph.mon.b.smithi153.stdout: self._sslobj.do_handshake() 2024-09-20T22:39:22.282 INFO:journalctl@ceph.mon.b.smithi153.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:39:22.282 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-mon[24571]: [20/Sep/2024:22:39:21] ENGINE Serving on http://172.21.15.153:8765 2024-09-20T22:39:22.282 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:21 smithi153 ceph-mon[24571]: [20/Sep/2024:22:39:21] ENGINE Bus STARTED 2024-09-20T22:39:22.848 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:22 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:22.529+0000 7fd5eabe6000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:39:22.848 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:22 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:22.847+0000 7fd5eabe6000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:39:23.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:22 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:22.984+0000 7fd5eabe6000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:39:23.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:23.082+0000 7fd5eabe6000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:39:23.467 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:23.189+0000 7fd5eabe6000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:39:23.467 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:23.280+0000 7fd5eabe6000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:39:23.467 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:23.435+0000 7fd5eabe6000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:39:23.864 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:23.534+0000 7fd5eabe6000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:39:23.864 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:23.634+0000 7fd5eabe6000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:39:23.864 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-mon[23720]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:39:23.865 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-mon[23720]: mgrmap e25: x(active, since 5s) 2024-09-20T22:39:23.865 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=error ts=2024-09-20T22:39:23.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.115:8443//api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused" 2024-09-20T22:39:23.865 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.115:8443//api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused" 2024-09-20T22:39:23.865 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-09-20T22:39:24.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:23 smithi153 ceph-mon[24571]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:39:24.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:23 smithi153 ceph-mon[24571]: mgrmap e25: x(active, since 5s) 2024-09-20T22:39:24.132 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-mon[28216]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:39:24.132 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-mon[28216]: mgrmap e25: x(active, since 5s) 2024-09-20T22:39:24.132 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:23.949+0000 7fd5eabe6000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:39:24.132 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:24 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:24.039+0000 7fd5eabe6000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:39:24.132 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:24 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:24.130+0000 7fd5eabe6000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:39:25.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:25.266+0000 7fd5eabe6000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:39:25.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:39:25] ENGINE Bus STARTING 2024-09-20T22:39:25.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: CherryPy Checker: 2024-09-20T22:39:25.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: The Application mounted at '' has an empty config. 2024-09-20T22:39:25.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: 2024-09-20T22:39:25.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:39:25] ENGINE Serving on http://:::9283 2024-09-20T22:39:25.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: [20/Sep/2024:22:39:25] ENGINE Bus STARTED 2024-09-20T22:39:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:39:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: Standby manager daemon y started 2024-09-20T22:39:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.115:0/3358915437' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-20T22:39:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.115:0/3358915437' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:39:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.115:0/3358915437' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-20T22:39:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.115:0/3358915437' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:39:26.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:26 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:26.337Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2024-09-20T22:39:26.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:39:26.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: Standby manager daemon y started 2024-09-20T22:39:26.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.115:0/3358915437' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-20T22:39:26.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.115:0/3358915437' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:39:26.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.115:0/3358915437' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-20T22:39:26.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.115:0/3358915437' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:39:26.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:39:26.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: Standby manager daemon y started 2024-09-20T22:39:26.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.115:0/3358915437' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-20T22:39:26.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.115:0/3358915437' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:39:26.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.115:0/3358915437' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-20T22:39:26.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.115:0/3358915437' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:39:26.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:26.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:26.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:26 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:27.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:27 smithi153 ceph-mon[24571]: Adjusting osd_memory_target on smithi115 to 2736M 2024-09-20T22:39:27.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:27 smithi153 ceph-mon[24571]: mgrmap e26: x(active, since 8s), standbys: y 2024-09-20T22:39:27.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:27 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:39:27.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:27 smithi115 ceph-mon[23720]: Adjusting osd_memory_target on smithi115 to 2736M 2024-09-20T22:39:27.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:27 smithi115 ceph-mon[23720]: mgrmap e26: x(active, since 8s), standbys: y 2024-09-20T22:39:27.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:27 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:39:27.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:27 smithi115 ceph-mon[28216]: Adjusting osd_memory_target on smithi115 to 2736M 2024-09-20T22:39:27.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:27 smithi115 ceph-mon[28216]: mgrmap e26: x(active, since 8s), standbys: y 2024-09-20T22:39:27.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:27 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:39:28.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:28 smithi153 ceph-mon[24571]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-20T22:39:28.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:28 smithi115 ceph-mon[23720]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-20T22:39:28.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:28 smithi115 ceph-mon[28216]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-20T22:39:29.723 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:39:29] "GET /metrics HTTP/1.1" 200 35164 "" "Prometheus/2.33.4" 2024-09-20T22:39:30.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:29 smithi153 ceph-mon[24571]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-20T22:39:30.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:29 smithi115 ceph-mon[23720]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-20T22:39:30.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:29 smithi115 ceph-mon[28216]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-20T22:39:31.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:31 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:39:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:39:32.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:31 smithi153 ceph-mon[24571]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T22:39:32.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:31 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:32.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:31 smithi115 ceph-mon[23720]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T22:39:32.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:31 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:32.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:31 smithi115 ceph-mon[28216]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T22:39:32.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:31 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:32.778 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:39:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:39:32.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.153:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.115:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:39:33.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:33 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=error ts=2024-09-20T22:39:33.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-09-20T22:39:33.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:33 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2024-09-20T22:39:33.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:33 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:33.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-09-20T22:39:34.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:33 smithi153 ceph-mon[24571]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:39:34.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:33 smithi115 ceph-mon[23720]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:39:34.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:33 smithi115 ceph-mon[28216]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:39:35.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:34 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:39:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:34 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:34 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:35.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:34 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:39:35.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:34 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:35.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:34 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:35.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:34 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:39:35.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:34 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:35.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:34 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:36.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:36 smithi115 ceph-mon[23720]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:39:36.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:36 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:36.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:36 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:36.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:36 smithi115 ceph-mon[28216]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:39:36.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:36 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:36.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:36 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:36.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:36 smithi153 ceph-mon[24571]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:39:36.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:36 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:36.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:36 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:38.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:38 smithi153 ceph-mon[24571]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:39:38.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:38 smithi115 ceph-mon[23720]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:39:38.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:38 smithi115 ceph-mon[28216]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:39:39.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:39:39] "GET /metrics HTTP/1.1" 200 38163 "" "Prometheus/2.33.4" 2024-09-20T22:39:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: Adjusting osd_memory_target on smithi153 to 2992M 2024-09-20T22:39:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:39:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:39:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:39 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: Adjusting osd_memory_target on smithi153 to 2992M 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:39.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: Adjusting osd_memory_target on smithi153 to 2992M 2024-09-20T22:39:39.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:39.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:39.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:39:39.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:39:39.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:39.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:39 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:40.404 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:40 smithi115 systemd[1]: Stopping Ceph node-exporter.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:39:40.904 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:40 smithi115 podman[56985]: 2024-09-20 22:39:40.502423528 +0000 UTC m=+0.125173161 container died 111c7bbee4de5f493d1ef9c9b72630e703c2717d63a1b590c540f0bcedb38409 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:39:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[23720]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[23720]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:39:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[23720]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:39:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[23720]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:39:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[23720]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:39:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[23720]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:39:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[23720]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:39:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:40.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[23720]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-20T22:39:40.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[23720]: Deploying daemon node-exporter.a on smithi115 2024-09-20T22:39:40.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[28216]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:40.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[28216]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:39:40.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[28216]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:39:40.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[28216]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:39:40.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[28216]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:39:40.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[28216]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:39:40.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[28216]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:39:40.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:40.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:40.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:40.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:40.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[28216]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-20T22:39:40.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:40 smithi115 ceph-mon[28216]: Deploying daemon node-exporter.a on smithi115 2024-09-20T22:39:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:40 smithi153 ceph-mon[24571]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:40 smithi153 ceph-mon[24571]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:39:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:40 smithi153 ceph-mon[24571]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:39:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:40 smithi153 ceph-mon[24571]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:39:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:40 smithi153 ceph-mon[24571]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:39:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:40 smithi153 ceph-mon[24571]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:39:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:40 smithi153 ceph-mon[24571]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:39:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:40 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:40 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:40 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:40 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:40 smithi153 ceph-mon[24571]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-20T22:39:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:40 smithi153 ceph-mon[24571]: Deploying daemon node-exporter.a on smithi115 2024-09-20T22:39:41.220 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:40 smithi115 podman[56985]: 2024-09-20 22:39:40.967514863 +0000 UTC m=+0.590264495 container cleanup 111c7bbee4de5f493d1ef9c9b72630e703c2717d63a1b590c540f0bcedb38409 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:39:41.220 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:40 smithi115 bash[56985]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a 2024-09-20T22:39:41.537 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:41 smithi115 podman[56996]: 2024-09-20 22:39:41.360326943 +0000 UTC m=+0.861408292 container remove 111c7bbee4de5f493d1ef9c9b72630e703c2717d63a1b590c540f0bcedb38409 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:39:41.537 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:41 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-20T22:39:41.537 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:41 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:39:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:39:41.904 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:41 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-20T22:39:41.904 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:41 smithi115 systemd[1]: Stopped Ceph node-exporter.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:39:41.904 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:41 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.a.service: Consumed 3.105s CPU time. 2024-09-20T22:39:41.904 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:41 smithi115 systemd[1]: Starting Ceph node-exporter.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:39:42.378 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:41 smithi115 bash[57123]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-20T22:39:42.640 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:42 smithi115 bash[57123]: Getting image source signatures 2024-09-20T22:39:42.641 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:42 smithi115 bash[57123]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-20T22:39:42.641 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:42 smithi115 bash[57123]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-20T22:39:42.641 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:42 smithi115 bash[57123]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-20T22:39:42.641 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:39:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:39:42.313Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.153:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.115:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:39:42.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:42 smithi115 ceph-mon[28216]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:42.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:42 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:42.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:42 smithi115 ceph-mon[23720]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:42.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:42 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:43.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:42 smithi153 ceph-mon[24571]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:42 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:43.904 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:43 smithi115 bash[57123]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-20T22:39:43.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:43 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=error ts=2024-09-20T22:39:43.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2024-09-20T22:39:43.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:43 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.115:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.115 because it doesn't contain any IP SANs" 2024-09-20T22:39:43.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:43 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=warn ts=2024-09-20T22:39:43.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-09-20T22:39:44.183 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:43 smithi115 bash[57123]: Writing manifest to image destination 2024-09-20T22:39:44.478 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 podman[57123]: 2024-09-20 22:39:44.157382452 +0000 UTC m=+2.296533331 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-20T22:39:44.478 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 podman[57123]: 2024-09-20 22:39:44.273675341 +0000 UTC m=+2.412826210 container create 807c51f95c4d867cc619f29e585a7edd598269dee82fd1aaad24c2f294ec77af (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:39:44.478 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 podman[57123]: 2024-09-20 22:39:44.477070485 +0000 UTC m=+2.616221356 container init 807c51f95c4d867cc619f29e585a7edd598269dee82fd1aaad24c2f294ec77af (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:39:44.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-mon[23720]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:44.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:44.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:44.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-mon[28216]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:44.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:44.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:44.905 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 podman[57123]: 2024-09-20 22:39:44.47995386 +0000 UTC m=+2.619104737 container start 807c51f95c4d867cc619f29e585a7edd598269dee82fd1aaad24c2f294ec77af (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:39:44.905 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.483Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-20T22:39:44.905 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.483Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-20T22:39:44.905 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.483Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-20T22:39:44.905 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T22:39:44.905 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-20T22:39:44.905 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-20T22:39:44.905 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-20T22:39:44.906 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=arp 2024-09-20T22:39:44.906 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-20T22:39:44.906 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-20T22:39:44.906 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-20T22:39:44.906 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-20T22:39:44.906 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-20T22:39:44.906 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-20T22:39:44.906 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-20T22:39:44.906 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-20T22:39:44.906 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=edac 2024-09-20T22:39:44.906 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-20T22:39:44.906 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-20T22:39:44.906 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-20T22:39:44.907 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-20T22:39:44.907 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-20T22:39:44.907 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-20T22:39:44.907 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-20T22:39:44.907 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-20T22:39:44.907 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-20T22:39:44.907 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-20T22:39:44.907 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-20T22:39:44.907 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-20T22:39:44.907 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-20T22:39:44.907 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-20T22:39:44.907 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-20T22:39:44.907 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-20T22:39:44.908 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=os 2024-09-20T22:39:44.908 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-20T22:39:44.908 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-20T22:39:44.908 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-20T22:39:44.908 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-20T22:39:44.908 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-20T22:39:44.908 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-20T22:39:44.908 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-20T22:39:44.908 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=stat 2024-09-20T22:39:44.908 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-20T22:39:44.908 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-20T22:39:44.909 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-20T22:39:44.909 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=time 2024-09-20T22:39:44.909 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-20T22:39:44.909 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=uname 2024-09-20T22:39:44.909 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-20T22:39:44.909 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-20T22:39:44.909 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.484Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-20T22:39:44.909 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.485Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-20T22:39:44.909 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a[57199]: ts=2024-09-20T22:39:44.485Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-20T22:39:44.909 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 bash[57123]: 807c51f95c4d867cc619f29e585a7edd598269dee82fd1aaad24c2f294ec77af 2024-09-20T22:39:44.909 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:39:44 smithi115 systemd[1]: Started Ceph node-exporter.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:39:45.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:44 smithi153 ceph-mon[24571]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:45.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:44 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:45.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:44 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:45.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:45 smithi115 ceph-mon[23720]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T22:39:45.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:45 smithi115 ceph-mon[23720]: Deploying daemon alertmanager.a on smithi115 2024-09-20T22:39:45.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:45 smithi115 ceph-mon[28216]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T22:39:45.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:45 smithi115 ceph-mon[28216]: Deploying daemon alertmanager.a on smithi115 2024-09-20T22:39:46.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:45 smithi153 ceph-mon[24571]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T22:39:46.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:45 smithi153 ceph-mon[24571]: Deploying daemon alertmanager.a on smithi115 2024-09-20T22:39:46.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:46 smithi115 ceph-mon[23720]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:46.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:46 smithi115 ceph-mon[28216]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:46.930 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:46 smithi153 ceph-mon[24571]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:48.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:48 smithi115 ceph-mon[23720]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:48.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:48 smithi115 ceph-mon[28216]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:49.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:48 smithi153 ceph-mon[24571]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:49.726 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:49 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:39:49] "GET /metrics HTTP/1.1" 200 38161 "" "Prometheus/2.33.4" 2024-09-20T22:39:50.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:49 smithi153 ceph-mon[24571]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:50.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:49 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:39:50.144 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:49 smithi115 ceph-mon[23720]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:50.144 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:49 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:39:50.144 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:49 smithi115 ceph-mon[28216]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:50.144 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:49 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:39:51.472 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:39:51.473 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:39:51 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:39:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:39:52.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:51 smithi115 ceph-mon[23720]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:52.062 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:51 smithi115 ceph-mon[23720]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:52.063 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:51 smithi115 ceph-mon[23720]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:52.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:51 smithi115 ceph-mon[28216]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:52.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:51 smithi115 ceph-mon[28216]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:52.063 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:51 smithi115 ceph-mon[28216]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:52.166 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:39:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4m) 26s ago 4m 27.8M - 0.23.0 ba2b418f427c cd90350c7e9a 2024-09-20T22:39:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (4m) 17s ago 4m 55.1M - dad864ee21e9 1549ac2de17e 2024-09-20T22:39:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (3m) 26s ago 3m 52.9M - 3.5 e1d6a67b021e e3ff3d2789f3 2024-09-20T22:39:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283 running (42s) 17s ago 9m 483M - 19.1.1-367-g9e2ac70b cc9251b744a3 e181bec15dad 2024-09-20T22:39:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:9283 running (10m) 26s ago 10m 167M - 17.2.0 e1d6a67b021e 8bc4016f6056 2024-09-20T22:39:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (10m) 26s ago 11m 49.3M 2048M 17.2.0 e1d6a67b021e 0b3311e89569 2024-09-20T22:39:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (9m) 17s ago 9m 43.4M 2048M 17.2.0 e1d6a67b021e 62b3afaddd26 2024-09-20T22:39:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (9m) 26s ago 9m 46.5M 2048M 17.2.0 e1d6a67b021e 3ee70f11001d 2024-09-20T22:39:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 starting - - - - 2024-09-20T22:39:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (5m) 17s ago 5m 14.7M - 1.3.1 1dbe0e931976 5717e9268a3a 2024-09-20T22:39:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (9m) 26s ago 9m 52.6M 2736M 17.2.0 e1d6a67b021e 06f9f63b0da2 2024-09-20T22:39:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (8m) 26s ago 8m 52.0M 2736M 17.2.0 e1d6a67b021e b68eba671ad6 2024-09-20T22:39:52.167 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (8m) 26s ago 8m 47.4M 2736M 17.2.0 e1d6a67b021e a368d2632c3f 2024-09-20T22:39:52.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (7m) 26s ago 7m 52.1M 2736M 17.2.0 e1d6a67b021e d4ef32545846 2024-09-20T22:39:52.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (7m) 17s ago 7m 53.9M 2992M 17.2.0 e1d6a67b021e e9316b3d7317 2024-09-20T22:39:52.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (7m) 17s ago 7m 49.5M 2992M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:39:52.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (6m) 17s ago 6m 47.7M 2992M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:39:52.168 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (6m) 17s ago 6m 50.7M 2992M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:39:52.168 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (4m) 17s ago 5m 65.9M - 2.33.4 514e6a882f6e 7dda2e0d2690 2024-09-20T22:39:52.168 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (4m) 26s ago 4m 72.6M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:39:52.168 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (4m) 17s ago 3m 72.8M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:39:52.168 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (3m) 26s ago 3m 71.5M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:39:52.168 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (3m) 17s ago 3m 69.3M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:39:52.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:51 smithi153 ceph-mon[24571]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:52.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:51 smithi153 ceph-mon[24571]: from='client.24778 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:39:52.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:51 smithi153 ceph-mon[24571]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:52.330 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:52 smithi115 systemd[1]: Stopping Ceph alertmanager.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:39:52.330 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:52 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[48068]: level=info ts=2024-09-20T22:39:52.308Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-09-20T22:39:52.579 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:39:52.579 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:39:52.579 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": {}, 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:39:52.580 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-09-20T22:39:52.581 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T22:39:52.581 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:39:52.581 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:39:52.581 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:52 smithi115 podman[57617]: 2024-09-20 22:39:52.328953806 +0000 UTC m=+0.110602197 container died cd90350c7e9aca6f9694f121c5a99f930a140b68f823089a1fed95a143fb97b8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:39:52.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:39:52 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:39:52.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.153:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.115:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:39:52.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:52 smithi115 podman[57617]: 2024-09-20 22:39:52.809377109 +0000 UTC m=+0.591025509 container cleanup cd90350c7e9aca6f9694f121c5a99f930a140b68f823089a1fed95a143fb97b8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:39:52.904 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:52 smithi115 bash[57617]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a 2024-09-20T22:39:52.941 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:39:52.941 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:39:52.941 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:39:52.941 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:39:52.941 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [], 2024-09-20T22:39:52.941 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "1/23 daemons upgraded", 2024-09-20T22:39:52.942 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "", 2024-09-20T22:39:52.942 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:39:52.942 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:39:53.171 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:52 smithi115 ceph-mon[23720]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:53.171 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:52 smithi115 ceph-mon[23720]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:53.171 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:52 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/180253154' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:39:53.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:52 smithi115 ceph-mon[28216]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:53.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:52 smithi115 ceph-mon[28216]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:53.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:52 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/180253154' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:39:53.172 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:53 smithi115 podman[57652]: 2024-09-20 22:39:53.069470365 +0000 UTC m=+0.739504849 container remove cd90350c7e9aca6f9694f121c5a99f930a140b68f823089a1fed95a143fb97b8 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:39:53.172 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:53 smithi115 podman[57652]: 2024-09-20 22:39:53.170109968 +0000 UTC m=+0.840144452 volume remove 8dbc72113590d272c87a93bbc8d945872930bdd16d18294408c994aebb589098 2024-09-20T22:39:53.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:52 smithi153 ceph-mon[24571]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:53.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:52 smithi153 ceph-mon[24571]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:53.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:52 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/180253154' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:39:53.361 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:39:53.505 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:53 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-20T22:39:53.505 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:53 smithi115 systemd[1]: Stopped Ceph alertmanager.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:39:53.505 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:53 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@alertmanager.a.service: Consumed 1.159s CPU time. 2024-09-20T22:39:53.778 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:39:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:39:53.506Z caller=notifier.go:526 level=error component=notifier alertmanager=http://172.21.15.115:9093/api/v2/alerts count=3 msg="Error sending alert" err="Post \"http://172.21.15.115:9093/api/v2/alerts\": dial tcp 172.21.15.115:9093: connect: connection refused" 2024-09-20T22:39:53.822 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:53 smithi115 systemd[1]: Starting Ceph alertmanager.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:39:53.823 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:53 smithi115 podman[57826]: 2024-09-20 22:39:53.613912034 +0000 UTC m=+0.015890296 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-20T22:39:53.823 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:53 smithi115 podman[57826]: 2024-09-20 22:39:53.729614839 +0000 UTC m=+0.131593129 volume create 8f9024ca1710b61dcda82645f5141ac30ab2b65da7ccfb99e59da0a7dbd86367 2024-09-20T22:39:53.823 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:53 smithi115 podman[57826]: 2024-09-20 22:39:53.821539727 +0000 UTC m=+0.223517973 container create 7b7065a4ae73c148820d85894e397aa051855b49f7420b00aa201e4854d9101e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:39:54.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:53 smithi115 ceph-mon[23720]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:54.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:53 smithi115 ceph-mon[23720]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:54.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:53 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/423804195' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:39:54.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:53 smithi115 ceph-mon[28216]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:54.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:53 smithi115 ceph-mon[28216]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:54.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:53 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/423804195' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:39:54.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:53 smithi153 ceph-mon[24571]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:54.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:53 smithi153 ceph-mon[24571]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:54.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:53 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/423804195' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:39:54.404 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:54 smithi115 podman[57826]: 2024-09-20 22:39:54.149844568 +0000 UTC m=+0.551822823 container init 7b7065a4ae73c148820d85894e397aa051855b49f7420b00aa201e4854d9101e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:39:54.404 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:54 smithi115 podman[57826]: 2024-09-20 22:39:54.152814013 +0000 UTC m=+0.554792268 container start 7b7065a4ae73c148820d85894e397aa051855b49f7420b00aa201e4854d9101e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:39:54.404 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:54 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:39:54.165Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-20T22:39:54.404 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:54 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:39:54.165Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-20T22:39:54.405 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:54 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:39:54.166Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.115 port=9094 2024-09-20T22:39:54.405 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:54 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:39:54.167Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-20T22:39:54.405 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:54 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:39:54.190Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T22:39:54.405 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:54 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:39:54.191Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T22:39:54.405 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:54 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:39:54.193Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-20T22:39:54.405 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:54 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:39:54.193Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-20T22:39:54.405 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:54 smithi115 bash[57826]: 7b7065a4ae73c148820d85894e397aa051855b49f7420b00aa201e4854d9101e 2024-09-20T22:39:54.405 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:54 smithi115 systemd[1]: Started Ceph alertmanager.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:39:55.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:55.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:55.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[23720]: Reconfiguring iscsi.foo.smithi115.hwguxp (dependencies changed)... 2024-09-20T22:39:55.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:39:55.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:39:55.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:55.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[23720]: Reconfiguring daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:39:55.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:55.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:55.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[28216]: Reconfiguring iscsi.foo.smithi115.hwguxp (dependencies changed)... 2024-09-20T22:39:55.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:39:55.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:39:55.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:55.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:55 smithi115 ceph-mon[28216]: Reconfiguring daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:39:55.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:55 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:55.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:55 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:39:55.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:55 smithi153 ceph-mon[24571]: Reconfiguring iscsi.foo.smithi115.hwguxp (dependencies changed)... 2024-09-20T22:39:55.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:55 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:39:55.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:55 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:39:55.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:55 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:55.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:55 smithi153 ceph-mon[24571]: Reconfiguring daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:39:56.465 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:56 smithi115 ceph-mon[23720]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:56.465 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:56 smithi115 ceph-mon[28216]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:56.465 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:39:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:39:56.168Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000521294s 2024-09-20T22:39:56.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:56 smithi153 ceph-mon[24571]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:39:57.623 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:39:57.290+0000 7fae6dbe1640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi115.hwguxp ... 2024-09-20T22:39:57.623 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: Creating ceph-iscsi config... 2024-09-20T22:39:57.623 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/iscsi-gateway.cfg 2024-09-20T22:39:57.623 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/tcmu-runner-entrypoint.sh 2024-09-20T22:39:57.623 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: Non-zero exit code 1 from systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp 2024-09-20T22:39:57.623 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: systemctl: stderr Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:57.623 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: systemctl: stderr See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:57.623 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: Traceback (most recent call last): 2024-09-20T22:39:57.623 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T22:39:57.624 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: return _run_code(code, main_globals, None, 2024-09-20T22:39:57.624 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T22:39:57.624 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: exec(code, run_globals) 2024-09-20T22:39:57.624 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5579, in 2024-09-20T22:39:57.624 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5567, in main 2024-09-20T22:39:57.624 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-20T22:39:57.624 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-20T22:39:57.624 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-20T22:39:57.624 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-20T22:39:57.624 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T22:39:57.624 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: RuntimeError: Failed command: systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp: Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:57.624 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:57.625 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: Traceback (most recent call last): 2024-09-20T22:39:57.625 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-20T22:39:57.625 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T22:39:57.625 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-20T22:39:57.625 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: return self.wait_async( 2024-09-20T22:39:57.625 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-20T22:39:57.625 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: return self.event_loop.get_result(coro, timeout) 2024-09-20T22:39:57.625 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T22:39:57.625 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: return future.result(timeout) 2024-09-20T22:39:57.626 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T22:39:57.626 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: return self.__get_result() 2024-09-20T22:39:57.626 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T22:39:57.626 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: raise self._exception 2024-09-20T22:39:57.626 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-20T22:39:57.626 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: out, err, code = await self._run_cephadm( 2024-09-20T22:39:57.626 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-20T22:39:57.626 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: raise OrchestratorError( 2024-09-20T22:39:57.626 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi115.hwguxp ... 2024-09-20T22:39:57.626 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: Creating ceph-iscsi config... 2024-09-20T22:39:57.626 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/iscsi-gateway.cfg 2024-09-20T22:39:57.626 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/tcmu-runner-entrypoint.sh 2024-09-20T22:39:57.626 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: Non-zero exit code 1 from systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp 2024-09-20T22:39:57.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: systemctl: stderr Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:57.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: systemctl: stderr See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:57.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: Traceback (most recent call last): 2024-09-20T22:39:57.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T22:39:57.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: return _run_code(code, main_globals, None, 2024-09-20T22:39:57.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T22:39:57.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: exec(code, run_globals) 2024-09-20T22:39:57.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5579, in 2024-09-20T22:39:57.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5567, in main 2024-09-20T22:39:57.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-20T22:39:57.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-20T22:39:57.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-20T22:39:57.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-20T22:39:57.628 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T22:39:57.628 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: RuntimeError: Failed command: systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp: Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:57.628 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:58.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:58 smithi153 ceph-mon[24571]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:58.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:58 smithi153 ceph-mon[24571]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi115.hwguxp ... 2024-09-20T22:39:58.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-09-20T22:39:58.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/iscsi-gateway.cfg 2024-09-20T22:39:58.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/tcmu-runner-entrypoint.sh 2024-09-20T22:39:58.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp 2024-09-20T22:39:58.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:58.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:58.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-09-20T22:39:58.529 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T22:39:58.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-09-20T22:39:58.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T22:39:58.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-09-20T22:39:58.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5579, in 2024-09-20T22:39:58.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5567, in main 2024-09-20T22:39:58.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-20T22:39:58.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-20T22:39:58.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-20T22:39:58.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-20T22:39:58.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T22:39:58.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp: Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:58.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:58.530 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.wait_async( 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: return future.result(timeout) 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.__get_result() 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: raise self._exception 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-20T22:39:58.531 INFO:journalctl@ceph.mon.b.smithi153.stdout: out, err, code = await self._run_cephadm( 2024-09-20T22:39:58.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-20T22:39:58.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: raise OrchestratorError( 2024-09-20T22:39:58.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi115.hwguxp ... 2024-09-20T22:39:58.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-09-20T22:39:58.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/iscsi-gateway.cfg 2024-09-20T22:39:58.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/tcmu-runner-entrypoint.sh 2024-09-20T22:39:58.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp 2024-09-20T22:39:58.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:58.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:58.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-09-20T22:39:58.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T22:39:58.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-09-20T22:39:58.532 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T22:39:58.533 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-09-20T22:39:58.533 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5579, in 2024-09-20T22:39:58.533 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5567, in main 2024-09-20T22:39:58.533 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-20T22:39:58.533 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-20T22:39:58.533 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-20T22:39:58.533 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-20T22:39:58.533 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T22:39:58.533 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp: Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:58.533 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:58.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:58 smithi153 ceph-mon[24571]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T22:39:58.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:58 smithi153 ceph-mon[24571]: Deploying daemon prometheus.a on smithi153 2024-09-20T22:39:58.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:58 smithi115 ceph-mon[23720]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:58.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:58 smithi115 ceph-mon[23720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi115.hwguxp ... 2024-09-20T22:39:58.655 INFO:journalctl@ceph.mon.a.smithi115.stdout: Creating ceph-iscsi config... 2024-09-20T22:39:58.655 INFO:journalctl@ceph.mon.a.smithi115.stdout: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/iscsi-gateway.cfg 2024-09-20T22:39:58.655 INFO:journalctl@ceph.mon.a.smithi115.stdout: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/tcmu-runner-entrypoint.sh 2024-09-20T22:39:58.655 INFO:journalctl@ceph.mon.a.smithi115.stdout: Non-zero exit code 1 from systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp 2024-09-20T22:39:58.655 INFO:journalctl@ceph.mon.a.smithi115.stdout: systemctl: stderr Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:58.655 INFO:journalctl@ceph.mon.a.smithi115.stdout: systemctl: stderr See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:58.655 INFO:journalctl@ceph.mon.a.smithi115.stdout: Traceback (most recent call last): 2024-09-20T22:39:58.655 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T22:39:58.655 INFO:journalctl@ceph.mon.a.smithi115.stdout: return _run_code(code, main_globals, None, 2024-09-20T22:39:58.655 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T22:39:58.655 INFO:journalctl@ceph.mon.a.smithi115.stdout: exec(code, run_globals) 2024-09-20T22:39:58.655 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5579, in 2024-09-20T22:39:58.655 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5567, in main 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: RuntimeError: Failed command: systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp: Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: Traceback (most recent call last): 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: return self.wait_async( 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-20T22:39:58.656 INFO:journalctl@ceph.mon.a.smithi115.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: return future.result(timeout) 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: return self.__get_result() 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: raise self._exception 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: out, err, code = await self._run_cephadm( 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: raise OrchestratorError( 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi115.hwguxp ... 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: Creating ceph-iscsi config... 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/iscsi-gateway.cfg 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/tcmu-runner-entrypoint.sh 2024-09-20T22:39:58.657 INFO:journalctl@ceph.mon.a.smithi115.stdout: Non-zero exit code 1 from systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: systemctl: stderr Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: systemctl: stderr See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: Traceback (most recent call last): 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: return _run_code(code, main_globals, None, 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: exec(code, run_globals) 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5579, in 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5567, in main 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-20T22:39:58.658 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T22:39:58.659 INFO:journalctl@ceph.mon.a.smithi115.stdout: RuntimeError: Failed command: systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp: Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:58.659 INFO:journalctl@ceph.mon.a.smithi115.stdout: See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:58.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:58 smithi115 ceph-mon[23720]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T22:39:58.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:58 smithi115 ceph-mon[23720]: Deploying daemon prometheus.a on smithi153 2024-09-20T22:39:58.659 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:58 smithi115 ceph-mon[28216]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:39:58.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:58 smithi115 ceph-mon[28216]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi115.hwguxp ... 2024-09-20T22:39:58.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: Creating ceph-iscsi config... 2024-09-20T22:39:58.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/iscsi-gateway.cfg 2024-09-20T22:39:58.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/tcmu-runner-entrypoint.sh 2024-09-20T22:39:58.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: Non-zero exit code 1 from systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp 2024-09-20T22:39:58.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: systemctl: stderr Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:58.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: systemctl: stderr See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:58.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2024-09-20T22:39:58.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T22:39:58.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: return _run_code(code, main_globals, None, 2024-09-20T22:39:58.660 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T22:39:58.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: exec(code, run_globals) 2024-09-20T22:39:58.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5579, in 2024-09-20T22:39:58.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5567, in main 2024-09-20T22:39:58.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-20T22:39:58.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-20T22:39:58.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-20T22:39:58.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-20T22:39:58.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T22:39:58.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: RuntimeError: Failed command: systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp: Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:58.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:58.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2024-09-20T22:39:58.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-20T22:39:58.661 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.wait_async( 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: return future.result(timeout) 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.__get_result() 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise self._exception 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: out, err, code = await self._run_cephadm( 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-20T22:39:58.662 INFO:journalctl@ceph.mon.c.smithi115.stdout: raise OrchestratorError( 2024-09-20T22:39:58.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi115.hwguxp ... 2024-09-20T22:39:58.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: Creating ceph-iscsi config... 2024-09-20T22:39:58.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/iscsi-gateway.cfg 2024-09-20T22:39:58.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/tcmu-runner-entrypoint.sh 2024-09-20T22:39:58.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: Non-zero exit code 1 from systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp 2024-09-20T22:39:58.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: systemctl: stderr Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:58.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: systemctl: stderr See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:58.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2024-09-20T22:39:58.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T22:39:58.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: return _run_code(code, main_globals, None, 2024-09-20T22:39:58.663 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T22:39:58.664 INFO:journalctl@ceph.mon.c.smithi115.stdout: exec(code, run_globals) 2024-09-20T22:39:58.664 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5579, in 2024-09-20T22:39:58.664 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5567, in main 2024-09-20T22:39:58.664 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-20T22:39:58.664 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-20T22:39:58.664 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-20T22:39:58.664 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-20T22:39:58.664 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T22:39:58.664 INFO:journalctl@ceph.mon.c.smithi115.stdout: RuntimeError: Failed command: systemctl restart ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp: Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:39:58.665 INFO:journalctl@ceph.mon.c.smithi115.stdout: See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:39:58.665 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:58 smithi115 ceph-mon[28216]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T22:39:58.665 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:58 smithi115 ceph-mon[28216]: Deploying daemon prometheus.a on smithi153 2024-09-20T22:39:59.729 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:39:59 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:39:59] "GET /metrics HTTP/1.1" 200 38161 "" "Prometheus/2.33.4" 2024-09-20T22:40:00.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:39:59 smithi153 ceph-mon[24571]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:39:59 smithi115 ceph-mon[23720]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:00.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:39:59 smithi115 ceph-mon[28216]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:01.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:00 smithi153 ceph-mon[24571]: overall HEALTH_OK 2024-09-20T22:40:01.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:00 smithi115 ceph-mon[23720]: overall HEALTH_OK 2024-09-20T22:40:01.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:00 smithi115 ceph-mon[28216]: overall HEALTH_OK 2024-09-20T22:40:01.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:01 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[23948]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:40:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-20T22:40:02.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:01 smithi153 ceph-mon[24571]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-20T22:40:02.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:01 smithi115 ceph-mon[23720]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-20T22:40:02.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:01 smithi115 ceph-mon[28216]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-20T22:40:02.528 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:02 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:40:02.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.153:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.115:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:40:04.173 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:03 smithi115 ceph-mon[23720]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-09-20T22:40:04.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:03 smithi115 ceph-mon[28216]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-09-20T22:40:04.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:03 smithi153 ceph-mon[24571]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-09-20T22:40:04.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:40:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:40:04.171Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004121207s 2024-09-20T22:40:05.166 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:04 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:05.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:04 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:05.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:04 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:06.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:05 smithi153 ceph-mon[24571]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-09-20T22:40:06.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:05 smithi115 ceph-mon[23720]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-09-20T22:40:06.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:05 smithi115 ceph-mon[28216]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-09-20T22:40:07.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:07 smithi153 ceph-mon[24571]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-09-20T22:40:07.987 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 systemd[1]: Stopping Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:40:08.083 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:07 smithi115 ceph-mon[23720]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-09-20T22:40:08.083 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:07 smithi115 ceph-mon[28216]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-09-20T22:40:08.262 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:40:07.985Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T22:40:08.262 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:40:07.985Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-09-20T22:40:08.262 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:40:07.985Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-09-20T22:40:08.262 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:40:07.985Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-09-20T22:40:08.262 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:40:07.985Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-09-20T22:40:08.263 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:40:07.985Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-09-20T22:40:08.263 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:40:07.985Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T22:40:08.263 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:40:07.985Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-09-20T22:40:08.263 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:40:07.985Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T22:40:08.263 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:40:07.986Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T22:40:08.263 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:40:07.986Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-09-20T22:40:08.263 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[41458]: ts=2024-09-20T22:40:07.986Z caller=main.go:1066 level=info msg="See you next time!" 2024-09-20T22:40:08.263 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:07 smithi153 podman[49166]: 2024-09-20 22:40:07.995503173 +0000 UTC m=+0.127809002 container died 7dda2e0d26909fa42a5383fb311694729e8bf27783e5678d7d8ed1a7f376c662 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:40:08.528 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:08 smithi153 podman[49166]: 2024-09-20 22:40:08.368090306 +0000 UTC m=+0.500396132 container cleanup 7dda2e0d26909fa42a5383fb311694729e8bf27783e5678d7d8ed1a7f376c662 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:40:08.528 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:08 smithi153 bash[49166]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T22:40:08.983 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:08 smithi153 podman[49181]: 2024-09-20 22:40:08.619354873 +0000 UTC m=+0.625568599 container remove 7dda2e0d26909fa42a5383fb311694729e8bf27783e5678d7d8ed1a7f376c662 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:40:08.983 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:08 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-20T22:40:08.983 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:08 smithi153 systemd[1]: Stopped Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:40:08.983 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:08 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@prometheus.a.service: Consumed 1.864s CPU time. 2024-09-20T22:40:08.983 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:08 smithi153 systemd[1]: Starting Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:40:09.279 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 podman[49307]: 2024-09-20 22:40:09.094211543 +0000 UTC m=+0.017430554 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-20T22:40:09.279 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 podman[49307]: 2024-09-20 22:40:09.220791539 +0000 UTC m=+0.144010554 container create dfa004a5353398d8a18af360e7da44e6cd97dd7dee95bc113d81864054a7abed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:40:09.733 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 podman[49307]: 2024-09-20 22:40:09.44772242 +0000 UTC m=+0.370941432 container init dfa004a5353398d8a18af360e7da44e6cd97dd7dee95bc113d81864054a7abed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:40:09.733 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 podman[49307]: 2024-09-20 22:40:09.450567695 +0000 UTC m=+0.373786720 container start dfa004a5353398d8a18af360e7da44e6cd97dd7dee95bc113d81864054a7abed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:40:09.733 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.475Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-20T22:40:09.733 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.475Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-20T22:40:09.733 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.475Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi153 (none))" 2024-09-20T22:40:09.733 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.475Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T22:40:09.733 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.475Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T22:40:09.733 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.476Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T22:40:09.733 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.477Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-20T22:40:09.733 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.477Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T22:40:09.733 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.477Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T22:40:09.734 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.480Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T22:40:09.734 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.480Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.601µs 2024-09-20T22:40:09.734 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.480Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T22:40:09.734 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.488Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-20T22:40:09.734 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.499Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-20T22:40:09.734 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.499Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-20T22:40:09.734 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.499Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.794µs wal_replay_duration=19.610362ms wbl_replay_duration=128ns total_replay_duration=19.647869ms 2024-09-20T22:40:09.734 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.502Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T22:40:09.734 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.502Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-20T22:40:09.734 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.502Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T22:40:09.734 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.523Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.629756ms db_storage=876ns remote_storage=1.159µs web_handler=427ns query_engine=560ns scrape=408.555µs scrape_sd=85.536µs notify=15.969µs notify_sd=25.25µs rules=20.755654ms tracing=7.207µs 2024-09-20T22:40:09.734 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.523Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-20T22:40:09.734 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:09.524Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T22:40:09.735 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 bash[49307]: dfa004a5353398d8a18af360e7da44e6cd97dd7dee95bc113d81864054a7abed 2024-09-20T22:40:09.735 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:09 smithi153 systemd[1]: Started Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:40:10.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-mon[24571]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T22:40:10.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/591593302' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:40:10.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2181598086' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2826853796"}]: dispatch 2024-09-20T22:40:10.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:10.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:10.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-mon[24571]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-20T22:40:10.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:09 smithi153 ceph-mon[24571]: Deploying daemon node-exporter.b on smithi153 2024-09-20T22:40:10.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[23720]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T22:40:10.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/591593302' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:40:10.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2181598086' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2826853796"}]: dispatch 2024-09-20T22:40:10.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:10.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:10.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[23720]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-20T22:40:10.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[23720]: Deploying daemon node-exporter.b on smithi153 2024-09-20T22:40:10.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[28216]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T22:40:10.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/591593302' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:40:10.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2181598086' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2826853796"}]: dispatch 2024-09-20T22:40:10.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:10.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:10.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[28216]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-20T22:40:10.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:09 smithi115 ceph-mon[28216]: Deploying daemon node-exporter.b on smithi153 2024-09-20T22:40:10.373 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:10 smithi153 systemd[1]: Stopping Ceph node-exporter.b for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:40:10.679 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:10 smithi153 podman[49489]: 2024-09-20 22:40:10.372059963 +0000 UTC m=+0.118321974 container died 5717e9268a3a6c2ac8a45b453a01b490189868675607c98ce070fb95c44b323c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:40:11.029 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:10 smithi153 podman[49489]: 2024-09-20 22:40:10.776919739 +0000 UTC m=+0.523181747 container cleanup 5717e9268a3a6c2ac8a45b453a01b490189868675607c98ce070fb95c44b323c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:40:11.029 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:10 smithi153 bash[49489]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b 2024-09-20T22:40:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:10 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2181598086' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2826853796"}]': finished 2024-09-20T22:40:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:10 smithi153 ceph-mon[24571]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T22:40:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:10 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1483468129' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/3218602138"}]: dispatch 2024-09-20T22:40:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:10 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2181598086' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2826853796"}]': finished 2024-09-20T22:40:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:10 smithi115 ceph-mon[23720]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T22:40:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:10 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1483468129' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/3218602138"}]: dispatch 2024-09-20T22:40:11.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:10 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2181598086' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2826853796"}]': finished 2024-09-20T22:40:11.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:10 smithi115 ceph-mon[28216]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T22:40:11.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:10 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1483468129' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/3218602138"}]: dispatch 2024-09-20T22:40:11.313 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:11 smithi153 podman[49502]: 2024-09-20 22:40:11.028191155 +0000 UTC m=+0.656659211 container remove 5717e9268a3a6c2ac8a45b453a01b490189868675607c98ce070fb95c44b323c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:40:11.313 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:11 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-20T22:40:11.313 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:11 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-20T22:40:11.313 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:11 smithi153 systemd[1]: Stopped Ceph node-exporter.b for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:40:11.313 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:11 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.b.service: Consumed 3.196s CPU time. 2024-09-20T22:40:11.623 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:11 smithi153 systemd[1]: Starting Ceph node-exporter.b for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:40:11.982 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:11 smithi153 bash[49639]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-20T22:40:11.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:11 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1483468129' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/3218602138"}]': finished 2024-09-20T22:40:11.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:11 smithi153 ceph-mon[24571]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T22:40:11.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:11 smithi153 ceph-mon[24571]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T22:40:11.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:11 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1719878152' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3218602138"}]: dispatch 2024-09-20T22:40:11.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:11 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3218602138"}]: dispatch 2024-09-20T22:40:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:11 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1483468129' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/3218602138"}]': finished 2024-09-20T22:40:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:11 smithi115 ceph-mon[23720]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T22:40:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:11 smithi115 ceph-mon[23720]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T22:40:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:11 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1719878152' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3218602138"}]: dispatch 2024-09-20T22:40:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:11 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3218602138"}]: dispatch 2024-09-20T22:40:12.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:11 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1483468129' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/3218602138"}]': finished 2024-09-20T22:40:12.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:11 smithi115 ceph-mon[28216]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T22:40:12.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:11 smithi115 ceph-mon[28216]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T22:40:12.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:11 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1719878152' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3218602138"}]: dispatch 2024-09-20T22:40:12.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:11 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3218602138"}]: dispatch 2024-09-20T22:40:12.278 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:11 smithi153 bash[49639]: Getting image source signatures 2024-09-20T22:40:12.279 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:11 smithi153 bash[49639]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-20T22:40:12.279 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:11 smithi153 bash[49639]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-20T22:40:12.279 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:11 smithi153 bash[49639]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-20T22:40:13.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:12 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3218602138"}]': finished 2024-09-20T22:40:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:12 smithi153 ceph-mon[24571]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T22:40:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:12 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/9324698' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4223966031"}]: dispatch 2024-09-20T22:40:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:12 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3218602138"}]': finished 2024-09-20T22:40:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:12 smithi115 ceph-mon[23720]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T22:40:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:12 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/9324698' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4223966031"}]: dispatch 2024-09-20T22:40:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:12 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/3218602138"}]': finished 2024-09-20T22:40:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:12 smithi115 ceph-mon[28216]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T22:40:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:12 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/9324698' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4223966031"}]: dispatch 2024-09-20T22:40:14.029 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:13 smithi153 bash[49639]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-20T22:40:14.029 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:13 smithi153 bash[49639]: Writing manifest to image destination 2024-09-20T22:40:14.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:13 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/9324698' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4223966031"}]': finished 2024-09-20T22:40:14.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:13 smithi153 ceph-mon[24571]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T22:40:14.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:13 smithi153 ceph-mon[24571]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T22:40:14.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:13 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1895069192' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1102177079"}]: dispatch 2024-09-20T22:40:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:13 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/9324698' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4223966031"}]': finished 2024-09-20T22:40:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:13 smithi115 ceph-mon[23720]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T22:40:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:13 smithi115 ceph-mon[23720]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T22:40:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:13 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1895069192' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1102177079"}]: dispatch 2024-09-20T22:40:14.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:13 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/9324698' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/4223966031"}]': finished 2024-09-20T22:40:14.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:13 smithi115 ceph-mon[28216]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T22:40:14.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:13 smithi115 ceph-mon[28216]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-20T22:40:14.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:13 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1895069192' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1102177079"}]: dispatch 2024-09-20T22:40:14.344 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 podman[49639]: 2024-09-20 22:40:13.98377663 +0000 UTC m=+2.454480823 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-20T22:40:14.344 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 podman[49639]: 2024-09-20 22:40:14.228841672 +0000 UTC m=+2.699545855 container create 3f057318ca363453e1220e6a9405f0bac57b29de1cf86494d260005de29eedd7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:40:14.774 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:40:14] "GET /metrics HTTP/1.1" 200 38162 "" "Prometheus/2.43.0" 2024-09-20T22:40:14.774 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 podman[49639]: 2024-09-20 22:40:14.472160141 +0000 UTC m=+2.942864330 container init 3f057318ca363453e1220e6a9405f0bac57b29de1cf86494d260005de29eedd7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:40:14.774 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 podman[49639]: 2024-09-20 22:40:14.47550134 +0000 UTC m=+2.946205524 container start 3f057318ca363453e1220e6a9405f0bac57b29de1cf86494d260005de29eedd7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:40:14.774 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.479Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-20T22:40:14.774 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.479Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-20T22:40:14.774 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.480Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-20T22:40:14.775 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.480Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-20T22:40:14.775 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.480Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-20T22:40:14.775 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.480Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T22:40:14.775 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-20T22:40:14.775 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=arp 2024-09-20T22:40:14.775 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-20T22:40:14.775 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-20T22:40:14.775 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-20T22:40:14.775 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-20T22:40:14.775 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-20T22:40:14.775 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-20T22:40:14.776 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-20T22:40:14.776 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-20T22:40:14.776 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=edac 2024-09-20T22:40:14.776 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-20T22:40:14.776 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-20T22:40:14.776 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-20T22:40:14.776 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-20T22:40:14.776 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-20T22:40:14.776 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-20T22:40:14.776 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-20T22:40:14.776 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-20T22:40:14.776 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=os 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-20T22:40:14.777 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=stat 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=time 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=uname 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.481Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.482Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b[49712]: ts=2024-09-20T22:40:14.482Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 bash[49639]: 3f057318ca363453e1220e6a9405f0bac57b29de1cf86494d260005de29eedd7 2024-09-20T22:40:14.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:40:14 smithi153 systemd[1]: Started Ceph node-exporter.b for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:40:15.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1895069192' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1102177079"}]': finished 2024-09-20T22:40:15.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-mon[24571]: osdmap e102: 8 total, 8 up, 8 in 2024-09-20T22:40:15.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1247112986' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2879365262"}]: dispatch 2024-09-20T22:40:15.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2879365262"}]: dispatch 2024-09-20T22:40:15.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:15.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:15.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-mon[24571]: Reconfiguring grafana.a (dependencies changed)... 2024-09-20T22:40:15.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-mon[24571]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T22:40:15.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:15.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:15.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:40:15.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:40:15.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:14 smithi153 ceph-mon[24571]: Reconfiguring daemon grafana.a on smithi153 2024-09-20T22:40:15.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1895069192' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1102177079"}]': finished 2024-09-20T22:40:15.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[23720]: osdmap e102: 8 total, 8 up, 8 in 2024-09-20T22:40:15.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1247112986' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2879365262"}]: dispatch 2024-09-20T22:40:15.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2879365262"}]: dispatch 2024-09-20T22:40:15.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:15.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:15.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[23720]: Reconfiguring grafana.a (dependencies changed)... 2024-09-20T22:40:15.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[23720]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T22:40:15.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:15.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:15.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:40:15.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:40:15.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[23720]: Reconfiguring daemon grafana.a on smithi153 2024-09-20T22:40:15.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1895069192' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1102177079"}]': finished 2024-09-20T22:40:15.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[28216]: osdmap e102: 8 total, 8 up, 8 in 2024-09-20T22:40:15.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1247112986' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2879365262"}]: dispatch 2024-09-20T22:40:15.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2879365262"}]: dispatch 2024-09-20T22:40:15.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:15.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:15.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[28216]: Reconfiguring grafana.a (dependencies changed)... 2024-09-20T22:40:15.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[28216]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T22:40:15.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:15.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:15.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:40:15.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:40:15.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:14 smithi115 ceph-mon[28216]: Reconfiguring daemon grafana.a on smithi153 2024-09-20T22:40:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:15 smithi115 ceph-mon[23720]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2879365262"}]': finished 2024-09-20T22:40:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:15 smithi115 ceph-mon[23720]: osdmap e103: 8 total, 8 up, 8 in 2024-09-20T22:40:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:15 smithi115 ceph-mon[23720]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:40:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:15 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3329000872' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1977701917"}]: dispatch 2024-09-20T22:40:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:15 smithi115 ceph-mon[28216]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2879365262"}]': finished 2024-09-20T22:40:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:15 smithi115 ceph-mon[28216]: osdmap e103: 8 total, 8 up, 8 in 2024-09-20T22:40:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:15 smithi115 ceph-mon[28216]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:40:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:15 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3329000872' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1977701917"}]: dispatch 2024-09-20T22:40:16.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:15 smithi153 ceph-mon[24571]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2879365262"}]': finished 2024-09-20T22:40:16.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:15 smithi153 ceph-mon[24571]: osdmap e103: 8 total, 8 up, 8 in 2024-09-20T22:40:16.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:15 smithi153 ceph-mon[24571]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:40:16.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:15 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3329000872' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1977701917"}]: dispatch 2024-09-20T22:40:16.912 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:16 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3329000872' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1977701917"}]': finished 2024-09-20T22:40:16.912 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:16 smithi153 ceph-mon[24571]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T22:40:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:16 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3329000872' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1977701917"}]': finished 2024-09-20T22:40:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:16 smithi115 ceph-mon[23720]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T22:40:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:16 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3329000872' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1977701917"}]': finished 2024-09-20T22:40:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:16 smithi115 ceph-mon[28216]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T22:40:18.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:17 smithi153 ceph-mon[24571]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:18.029 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:17 smithi153 systemd[1]: Stopping Ceph grafana.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:40:18.029 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[40681]: t=2024-09-20T22:40:17+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-09-20T22:40:18.029 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:17 smithi153 podman[49888]: 2024-09-20 22:40:17.955511336 +0000 UTC m=+0.115877680 container died 1549ac2de17eb6f2c8b988043ee10f0427eebdcbf25fdb2f2fcbf764552feeaf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, maintainer=Paul Cuzner , name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-09-20T22:40:18.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:17 smithi115 ceph-mon[23720]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:18.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:17 smithi115 ceph-mon[28216]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:18.660 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:18 smithi153 podman[49888]: 2024-09-20 22:40:18.41166415 +0000 UTC m=+0.572030540 container cleanup 1549ac2de17eb6f2c8b988043ee10f0427eebdcbf25fdb2f2fcbf764552feeaf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, name=ubi8, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, distribution-scope=public, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5) 2024-09-20T22:40:18.660 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:18 smithi153 bash[49888]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a 2024-09-20T22:40:18.661 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:18 smithi153 bash[49914]: Error: no container with name or ID "ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana.a" found: no such container 2024-09-20T22:40:18.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:18 smithi115 ceph-mon[23720]: from='client.34874 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:18.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:18 smithi115 ceph-mon[28216]: from='client.34874 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:19.020 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:18 smithi153 ceph-mon[24571]: from='client.34874 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:19.020 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:18.658Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:40:19.021 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:18 smithi153 podman[49900]: 2024-09-20 22:40:18.788184051 +0000 UTC m=+0.827600935 container remove 1549ac2de17eb6f2c8b988043ee10f0427eebdcbf25fdb2f2fcbf764552feeaf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a, version=8.5, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., release=236.1648460182, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, description=Ceph Grafana Container, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container) 2024-09-20T22:40:19.021 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:18 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@grafana.a.service: Deactivated successfully. 2024-09-20T22:40:19.021 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:18 smithi153 systemd[1]: Stopped Ceph grafana.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:40:19.021 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:18 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@grafana.a.service: Consumed 2.436s CPU time. 2024-09-20T22:40:19.021 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:18 smithi153 systemd[1]: Starting Ceph grafana.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:40:19.276 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 podman[49977]: 2024-09-20 22:40:18.921321079 +0000 UTC m=+0.015715930 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-09-20T22:40:19.276 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 podman[49977]: 2024-09-20 22:40:19.039021704 +0000 UTC m=+0.133416547 container create 568386e3ebfd2c688abb1ec4721008a0b561781daf551ae37817f38d2118b530 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.openshift.expose-services=, io.buildah.version=1.24.2, vcs-type=git, architecture=x86_64, description=Ceph Grafana Container, maintainer=Paul Cuzner , distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182) 2024-09-20T22:40:19.276 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 podman[49977]: 2024-09-20 22:40:19.274935216 +0000 UTC m=+0.369330062 container init 568386e3ebfd2c688abb1ec4721008a0b561781daf551ae37817f38d2118b530 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, description=Ceph Grafana Container, maintainer=Paul Cuzner , name=ubi8) 2024-09-20T22:40:19.528 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:19] ENGINE Bus STOPPING 2024-09-20T22:40:19.529 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 podman[49977]: 2024-09-20 22:40:19.278132131 +0000 UTC m=+0.372526958 container start 568386e3ebfd2c688abb1ec4721008a0b561781daf551ae37817f38d2118b530 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, distribution-scope=public, name=ubi8, architecture=x86_64, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, maintainer=Paul Cuzner , vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-09-20T22:40:19.529 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-09-20T22:40:19.529 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-09-20T22:40:19.529 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-09-20T22:40:19.530 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-09-20T22:40:19.530 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-20T22:40:19.530 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-20T22:40:19.530 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-20T22:40:19.530 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-20T22:40:19.530 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-09-20T22:40:19.531 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-09-20T22:40:19.531 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-09-20T22:40:19.531 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-09-20T22:40:19.531 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-09-20T22:40:19.531 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="App mode production" logger=settings 2024-09-20T22:40:19.531 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-09-20T22:40:19.531 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-20T22:40:19.531 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-09-20T22:40:19.531 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=494.926µs 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 bash[49977]: 568386e3ebfd2c688abb1ec4721008a0b561781daf551ae37817f38d2118b530 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 systemd[1]: Started Ceph grafana.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-09-20T22:40:19.532 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:40:19+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-09-20T22:40:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:40:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:40:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi115.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T22:40:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi115.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T22:40:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-mon[24571]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T22:40:20.031 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:40:20.031 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:19] ENGINE Bus STOPPED 2024-09-20T22:40:20.031 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:19] ENGINE Bus STARTING 2024-09-20T22:40:20.031 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:19] ENGINE Serving on http://:::9283 2024-09-20T22:40:20.031 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:19] ENGINE Bus STARTED 2024-09-20T22:40:20.031 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:19] ENGINE Bus STOPPING 2024-09-20T22:40:20.031 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:40:20.031 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:19] ENGINE Bus STOPPED 2024-09-20T22:40:20.031 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:19] ENGINE Bus STARTING 2024-09-20T22:40:20.031 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:19] ENGINE Serving on http://:::9283 2024-09-20T22:40:20.031 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:19] ENGINE Bus STARTED 2024-09-20T22:40:20.032 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:19] ENGINE Bus STOPPING 2024-09-20T22:40:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:40:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:40:20.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi115.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi115.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T22:40:20.113 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[28216]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T22:40:20.114 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:20.114 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:20.114 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.114 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.114 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:40:20.114 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:40:20.114 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi115.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T22:40:20.114 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi115.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T22:40:20.114 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.114 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:40:20.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:40:20.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T22:40:20.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T22:40:20.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:40:20.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:40:20.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T22:40:20.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T22:40:20.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:20.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:20.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T22:40:20.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:19 smithi115 ceph-mon[23720]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-20T22:40:20.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:40:20.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:20] ENGINE Bus STOPPED 2024-09-20T22:40:20.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:20] ENGINE Bus STARTING 2024-09-20T22:40:20.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:20] ENGINE Serving on http://:::9283 2024-09-20T22:40:20.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:20] ENGINE Bus STARTED 2024-09-20T22:40:22.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:21 smithi153 ceph-mon[24571]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:40:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:21 smithi153 ceph-mon[24571]: Upgrade: Updating mgr.y 2024-09-20T22:40:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:21 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:21 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:40:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:21 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:40:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:21 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:40:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:21 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:21 smithi153 ceph-mon[24571]: Deploying daemon mgr.y on smithi115 2024-09-20T22:40:22.286 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[23720]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:40:22.286 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[23720]: Upgrade: Updating mgr.y 2024-09-20T22:40:22.286 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:22.286 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:40:22.286 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:40:22.286 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:40:22.286 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:22.286 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[23720]: Deploying daemon mgr.y on smithi115 2024-09-20T22:40:22.286 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[28216]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:40:22.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[28216]: Upgrade: Updating mgr.y 2024-09-20T22:40:22.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:22.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:40:22.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:40:22.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:40:22.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:22.287 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:21 smithi115 ceph-mon[28216]: Deploying daemon mgr.y on smithi115 2024-09-20T22:40:22.866 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:22 smithi115 systemd[1]: Stopping Ceph mgr.y for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:40:23.151 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:22 smithi115 podman[58760]: 2024-09-20 22:40:22.864658816 +0000 UTC m=+0.140559440 container died 8bc4016f60564c377027c6313f699beb3eaa9ede83bc60263385958b64fea915 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., name=centos-stream, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.buildah.version=1.19.8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-09-20T22:40:23.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:23 smithi115 podman[58760]: 2024-09-20 22:40:23.31641058 +0000 UTC m=+0.592311188 container cleanup 8bc4016f60564c377027c6313f699beb3eaa9ede83bc60263385958b64fea915 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y, io.openshift.tags=base centos centos-stream, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., distribution-scope=public, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git) 2024-09-20T22:40:23.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:23 smithi115 bash[58760]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y 2024-09-20T22:40:23.731 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:40:23.778 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:23.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:40:23.904 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:23 smithi115 podman[58771]: 2024-09-20 22:40:23.634653143 +0000 UTC m=+0.766095188 container remove 8bc4016f60564c377027c6313f699beb3eaa9ede83bc60263385958b64fea915 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, distribution-scope=public, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, release=754, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True) 2024-09-20T22:40:23.904 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:23 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-20T22:40:23.904 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:23 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-20T22:40:23.904 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:23 smithi115 systemd[1]: Stopped Ceph mgr.y for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:40:23.904 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:23 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.y.service: Consumed 54.399s CPU time. 2024-09-20T22:40:24.236 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:24 smithi115 systemd[1]: Starting Ceph mgr.y for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:40:24.237 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:23 smithi115 ceph-mon[23720]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:24.237 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:23 smithi115 ceph-mon[23720]: from='client.25183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:24.237 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:23 smithi115 ceph-mon[28216]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:24.237 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:23 smithi115 ceph-mon[28216]: from='client.25183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:24.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:23 smithi153 ceph-mon[24571]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:24.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:23 smithi153 ceph-mon[24571]: from='client.25183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:24.430 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:40:24.430 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 starting - - - - 2024-09-20T22:40:24.430 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 starting - - - - 2024-09-20T22:40:24.430 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (4m) 59s ago 4m 52.9M - 3.5 e1d6a67b021e e3ff3d2789f3 2024-09-20T22:40:24.430 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283 running (75s) 49s ago 9m 483M - 19.1.1-367-g9e2ac70b cc9251b744a3 e181bec15dad 2024-09-20T22:40:24.430 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:9283 running (11m) 59s ago 11m 167M - 17.2.0 e1d6a67b021e 8bc4016f6056 2024-09-20T22:40:24.430 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (11m) 59s ago 11m 49.3M 2048M 17.2.0 e1d6a67b021e 0b3311e89569 2024-09-20T22:40:24.430 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (10m) 49s ago 10m 43.4M 2048M 17.2.0 e1d6a67b021e 62b3afaddd26 2024-09-20T22:40:24.430 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (10m) 59s ago 10m 46.5M 2048M 17.2.0 e1d6a67b021e 3ee70f11001d 2024-09-20T22:40:24.430 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 starting - - - - 2024-09-20T22:40:24.430 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 starting - - - - 2024-09-20T22:40:24.430 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (9m) 59s ago 9m 52.6M 2736M 17.2.0 e1d6a67b021e 06f9f63b0da2 2024-09-20T22:40:24.431 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (9m) 59s ago 9m 52.0M 2736M 17.2.0 e1d6a67b021e b68eba671ad6 2024-09-20T22:40:24.431 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (8m) 59s ago 8m 47.4M 2736M 17.2.0 e1d6a67b021e a368d2632c3f 2024-09-20T22:40:24.431 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (8m) 59s ago 8m 52.1M 2736M 17.2.0 e1d6a67b021e d4ef32545846 2024-09-20T22:40:24.431 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (8m) 49s ago 7m 53.9M 2992M 17.2.0 e1d6a67b021e e9316b3d7317 2024-09-20T22:40:24.431 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (7m) 49s ago 7m 49.5M 2992M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:40:24.431 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (7m) 49s ago 7m 47.7M 2992M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:40:24.431 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (6m) 49s ago 6m 50.7M 2992M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:40:24.431 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 starting - - - - 2024-09-20T22:40:24.431 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (4m) 59s ago 4m 72.6M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:40:24.431 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (4m) 49s ago 4m 72.8M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:40:24.431 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (4m) 59s ago 4m 71.5M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:40:24.431 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (4m) 49s ago 4m 69.3M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:40:24.491 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:24 smithi115 podman[58947]: 2024-09-20 22:40:24.138069488 +0000 UTC m=+0.016687493 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:40:24.491 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:24 smithi115 podman[58947]: 2024-09-20 22:40:24.235552868 +0000 UTC m=+0.114170858 container create 07b943d2934a7a3c2abdfc8ac3bc9e2469d678ab0c32fc25a1c4ddd7fe164db8 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T22:40:24.492 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:24 smithi115 podman[58947]: 2024-09-20 22:40:24.463357247 +0000 UTC m=+0.341975247 container init 07b943d2934a7a3c2abdfc8ac3bc9e2469d678ab0c32fc25a1c4ddd7fe164db8 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-20T22:40:24.492 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:24 smithi115 podman[58947]: 2024-09-20 22:40:24.466214581 +0000 UTC m=+0.344832581 container start 07b943d2934a7a3c2abdfc8ac3bc9e2469d678ab0c32fc25a1c4ddd7fe164db8 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:40:24.839 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:24 smithi115 ceph-mgr[58989]: -- 172.21.15.115:0/350031690 <== mon.1 v2:172.21.15.115:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fcddd570e0 con 0x55fcddd34800 2024-09-20T22:40:24.839 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:24 smithi115 bash[58947]: 07b943d2934a7a3c2abdfc8ac3bc9e2469d678ab0c32fc25a1c4ddd7fe164db8 2024-09-20T22:40:24.839 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:24 smithi115 systemd[1]: Started Ceph mgr.y for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:40:24.839 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:24 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:24.616+0000 7f8033750040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:40:24.840 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:24 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:24.677+0000 7f8033750040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:40:24.885 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:40:24.885 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:40:24.885 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-09-20T22:40:24.885 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:40:24.886 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:40:24.886 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-09-20T22:40:24.886 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T22:40:24.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:40:24.886 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:40:24.886 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T22:40:24.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:40:24.886 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": {}, 2024-09-20T22:40:24.886 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:40:24.886 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:40:24.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:40:24.886 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:40:24.887 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-09-20T22:40:24.887 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T22:40:24.887 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:40:24.887 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:40:25.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:24 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:40:24] "GET /metrics HTTP/1.1" 200 38153 "" "Prometheus/2.43.0" 2024-09-20T22:40:25.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:24 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:24.836+0000 7f8033750040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:40:25.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:24 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:24.919+0000 7f8033750040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:40:25.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:25.084+0000 7f8033750040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:40:25.262 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:40:25.263 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:40:25.263 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:40:25.263 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:40:25.263 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [], 2024-09-20T22:40:25.263 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "", 2024-09-20T22:40:25.263 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading mgr daemons", 2024-09-20T22:40:25.263 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:40:25.263 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:40:25.542 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:25.237+0000 7f8033750040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:40:25.742 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:40:25.872 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:25.594+0000 7f8033750040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:40:25.872 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:25.665+0000 7f8033750040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:40:25.872 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[23720]: from='client.25189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:25.872 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[23720]: from='client.25195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:25.872 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:25.872 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:25.873 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/1969968177' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:40:25.873 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:25.873 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:25.873 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:25.873 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:25.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[28216]: from='client.25189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:25.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[28216]: from='client.25195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:25.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:25.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:25.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/1969968177' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:40:25.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:25.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:25.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:25.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:26.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:25 smithi153 ceph-mon[24571]: from='client.25189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:26.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:25 smithi153 ceph-mon[24571]: from='client.25195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:26.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:25 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:26.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:25 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:26.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:25 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/1969968177' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:40:26.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:25 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:26.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:25 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:26.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:25 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:26.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:25 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:26.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:25.869+0000 7f8033750040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:40:26.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:26 smithi115 ceph-mon[28216]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:40:26.904 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:26 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:26.827+0000 7f8033750040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:40:26.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:26 smithi115 ceph-mon[23720]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:40:26.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:26 smithi115 ceph-mon[23720]: from='client.25213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:26.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:26 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3115819523' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:40:26.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:26 smithi115 ceph-mon[28216]: from='client.25213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:26.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:26 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3115819523' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:40:27.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:26 smithi153 ceph-mon[24571]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:40:27.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:26 smithi153 ceph-mon[24571]: from='client.25213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:27.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:26 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3115819523' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:40:27.235 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:26 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:26.902+0000 7f8033750040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:40:27.927 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:27 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:27.659+0000 7f8033750040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:40:27.927 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:27 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:27.782+0000 7f8033750040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:40:28.180 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:27 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:27.925+0000 7f8033750040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:40:28.181 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:27 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:27.991+0000 7f8033750040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:40:28.181 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:28 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:28.145+0000 7f8033750040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T22:40:28.442 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:28 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:28.215+0000 7f8033750040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:40:28.442 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:28 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:28.275+0000 7f8033750040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:40:28.442 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:28 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:28.440+0000 7f8033750040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:40:28.737 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:28 smithi115 ceph-mon[23720]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-20T22:40:28.738 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:28 smithi115 ceph-mon[28216]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-20T22:40:28.738 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:28 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:28.519+0000 7f8033750040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:40:28.738 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:28 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:28.588+0000 7f8033750040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:40:28.738 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:28 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:28.673+0000 7f8033750040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:40:28.738 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:28 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:28.735+0000 7f8033750040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:40:28.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:28 smithi153 ceph-mon[24571]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-09-20T22:40:28.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:28.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:40:29.001 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:28 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:28.801+0000 7f8033750040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:40:29.639 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:29.622+0000 7f8033750040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:40:29.639 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:40:29] ENGINE Bus STARTING 2024-09-20T22:40:29.639 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: CherryPy Checker: 2024-09-20T22:40:29.639 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: The Application mounted at '' has an empty config. 2024-09-20T22:40:29.639 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:29.904 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:40:29] ENGINE Serving on http://:::9283 2024-09-20T22:40:29.904 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:40:29] ENGINE Bus STARTED 2024-09-20T22:40:29.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[23720]: from='client.34874 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:29.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:29.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:29.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[23720]: Standby manager daemon y restarted 2024-09-20T22:40:29.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[23720]: Standby manager daemon y started 2024-09-20T22:40:29.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-20T22:40:29.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:40:29.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-20T22:40:29.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:40:29.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[28216]: from='client.34874 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:29.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:29.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:29.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[28216]: Standby manager daemon y restarted 2024-09-20T22:40:29.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[28216]: Standby manager daemon y started 2024-09-20T22:40:29.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-20T22:40:29.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:40:29.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-20T22:40:29.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:29 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:40:30.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:29 smithi153 ceph-mon[24571]: from='client.34874 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:40:30.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:29 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:30.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:29 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:30.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:29 smithi153 ceph-mon[24571]: Standby manager daemon y restarted 2024-09-20T22:40:30.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:29 smithi153 ceph-mon[24571]: Standby manager daemon y started 2024-09-20T22:40:30.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:29 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-20T22:40:30.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:29 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:40:30.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:29 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-20T22:40:30.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:29 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:40:30.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:30 smithi115 ceph-mon[23720]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:30.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:30 smithi115 ceph-mon[23720]: mgrmap e27: x(active, since 71s), standbys: y 2024-09-20T22:40:30.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:30 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:30.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:30 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:30.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:30 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:30.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:30 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:30.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:30 smithi115 ceph-mon[28216]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:30.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:30 smithi115 ceph-mon[28216]: mgrmap e27: x(active, since 71s), standbys: y 2024-09-20T22:40:30.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:30 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:30.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:30 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:30.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:30 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:30.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:30 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:31.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:30 smithi153 ceph-mon[24571]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:31.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:30 smithi153 ceph-mon[24571]: mgrmap e27: x(active, since 71s), standbys: y 2024-09-20T22:40:31.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:30 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:31.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:30 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:31.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:30 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:31.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:30 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:32.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:32 smithi115 ceph-mon[23720]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:32.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:32 smithi115 ceph-mon[28216]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:33.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:32 smithi153 ceph-mon[24571]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:33.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:33.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:40:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[23720]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:40:34.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:40:34.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:34.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[28216]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:40:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:40:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:34 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:35.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:34 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:40:34] "GET /metrics HTTP/1.1" 200 38162 "" "Prometheus/2.43.0" 2024-09-20T22:40:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:34 smithi153 ceph-mon[24571]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:34 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:34 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:34 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:34 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:34 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:34 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:40:35.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:34 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:40:35.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:34 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:35.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:34 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:35.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:35 smithi115 ceph-mon[23720]: Reconfiguring iscsi.foo.smithi115.hwguxp (dependencies changed)... 2024-09-20T22:40:35.789 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:35 smithi115 ceph-mon[23720]: Reconfiguring daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:40:35.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:35 smithi115 ceph-mon[28216]: Reconfiguring iscsi.foo.smithi115.hwguxp (dependencies changed)... 2024-09-20T22:40:35.790 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:35 smithi115 ceph-mon[28216]: Reconfiguring daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:40:36.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:35 smithi153 ceph-mon[24571]: Reconfiguring iscsi.foo.smithi115.hwguxp (dependencies changed)... 2024-09-20T22:40:36.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:35 smithi153 ceph-mon[24571]: Reconfiguring daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:40:36.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:36 smithi115 ceph-mon[23720]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:36.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:36 smithi115 ceph-mon[28216]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:37.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:36 smithi153 ceph-mon[24571]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:40:38.879 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:40:38] ENGINE Bus STOPPING 2024-09-20T22:40:38.879 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:40:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:40:38.880 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:40:38] ENGINE Bus STOPPED 2024-09-20T22:40:38.880 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[23720]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:38.880 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:38.880 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:38.880 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:40:38.880 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:40:38.881 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:40:38.881 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[23720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:40:38.881 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:38.881 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[23720]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T22:40:38.881 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T22:40:38.881 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[23720]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T22:40:38.881 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/2592884453' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:40:38.881 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[28216]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:38.881 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:38.881 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:38.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:40:38.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:40:38.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:40:38.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[28216]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:40:38.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:38.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[28216]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T22:40:38.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T22:40:38.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[28216]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T22:40:38.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2592884453' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:40:38.935 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:38.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:40:38.935 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-mon[24571]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:40:38.935 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:38.935 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' 2024-09-20T22:40:38.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:40:38.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:40:38.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:40:38.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-mon[24571]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:40:38.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:38.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-mon[24571]: from='mgr.34793 172.21.15.153:0/479091960' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T22:40:38.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-20T22:40:38.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-mon[24571]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T22:40:38.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2592884453' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:40:38.936 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:38.705+0000 7faed90d5640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-20T22:40:38.936 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: ignoring --setuser ceph since I am not root 2024-09-20T22:40:38.937 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: ignoring --setgroup ceph since I am not root 2024-09-20T22:40:38.937 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-mgr[45636]: -- 172.21.15.153:0/1494122153 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ea5dcb0e0 con 0x562ea5da8800 2024-09-20T22:40:38.937 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:38.862+0000 7f6575cf5040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:40:38.937 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:38.933+0000 7f6575cf5040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:40:39.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:40:38] ENGINE Bus STARTING 2024-09-20T22:40:39.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:40:38] ENGINE Serving on http://:::9283 2024-09-20T22:40:39.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:38 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:40:38] ENGINE Bus STARTED 2024-09-20T22:40:39.243 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:39.097+0000 7f6575cf5040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:40:39.243 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:39.171+0000 7f6575cf5040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:40:39.517 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:39.365+0000 7f6575cf5040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:40:39.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:39.514+0000 7f6575cf5040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:40:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.34793 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-20T22:40:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: mgrmap e28: y(active, starting, since 0.952492s) 2024-09-20T22:40:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:40:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:40:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:40:39.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:40:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:40:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:40:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:40:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:40:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:39.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:40:39.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:40:39.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:40:39.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:40:39.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:40:39.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: Manager daemon y is now available 2024-09-20T22:40:39.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:39.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:40:39.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:40:40.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.34793 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-20T22:40:40.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: mgrmap e28: y(active, starting, since 0.952492s) 2024-09-20T22:40:40.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:40:40.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:40:40.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:40:40.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:40:40.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:40:40.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:40:40.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:40:40.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:40:40.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:40.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:40.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:40:40.019 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:40:40.020 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:40:40.020 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:40:40.020 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:40:40.020 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: Manager daemon y is now available 2024-09-20T22:40:40.020 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:40.020 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:40:40.020 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:40:40.020 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.34793 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-20T22:40:40.020 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: mgrmap e28: y(active, starting, since 0.952492s) 2024-09-20T22:40:40.020 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:40:40.020 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:40:40.020 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: Manager daemon y is now available 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:40.021 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:40:40.022 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:39 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:40:40.121 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:39.858+0000 7f6575cf5040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:40:40.122 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:39.923+0000 7f6575cf5040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:40:40.122 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:40 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:40.118+0000 7f6575cf5040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:40:40.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:40.082+0000 7f7f8ff4a640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:22:40:40] ENGINE Error in HTTPServer.serve 2024-09-20T22:40:40.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: Traceback (most recent call last): 2024-09-20T22:40:40.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:40:40.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: self._connections.run(self.expiration_interval) 2024-09-20T22:40:40.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:40:40.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: self._run(expiration_interval) 2024-09-20T22:40:40.406 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:40:40.406 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:40:40.406 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:40:40.406 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:40:40.406 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:40:40.407 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: s = self.context.wrap_socket( 2024-09-20T22:40:40.407 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:40:40.407 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: return self.sslsocket_class._create( 2024-09-20T22:40:40.407 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:40:40.407 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: self.do_handshake() 2024-09-20T22:40:40.408 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:40:40.408 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: self._sslobj.do_handshake() 2024-09-20T22:40:40.408 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:40:40.408 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:40:40.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-mon[23720]: mgrmap e29: y(active, since 1.98694s) 2024-09-20T22:40:40.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-mon[23720]: [20/Sep/2024:22:40:39] ENGINE Bus STARTING 2024-09-20T22:40:40.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-mon[23720]: [20/Sep/2024:22:40:40] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:40:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-mon[23720]: [20/Sep/2024:22:40:40] ENGINE Error in HTTPServer.serve 2024-09-20T22:40:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: Traceback (most recent call last): 2024-09-20T22:40:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:40:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:40:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:40:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: self._run(expiration_interval) 2024-09-20T22:40:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:40:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:40:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:40:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:40:40.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:40:40.906 INFO:journalctl@ceph.mon.a.smithi115.stdout: s = self.context.wrap_socket( 2024-09-20T22:40:40.906 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:40:40.906 INFO:journalctl@ceph.mon.a.smithi115.stdout: return self.sslsocket_class._create( 2024-09-20T22:40:40.906 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:40:40.906 INFO:journalctl@ceph.mon.a.smithi115.stdout: self.do_handshake() 2024-09-20T22:40:40.906 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:40:40.906 INFO:journalctl@ceph.mon.a.smithi115.stdout: self._sslobj.do_handshake() 2024-09-20T22:40:40.906 INFO:journalctl@ceph.mon.a.smithi115.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:40:40.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-mon[23720]: [20/Sep/2024:22:40:40] ENGINE Serving on http://172.21.15.115:8765 2024-09-20T22:40:40.907 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-mon[23720]: [20/Sep/2024:22:40:40] ENGINE Bus STARTED 2024-09-20T22:40:40.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-mon[28216]: mgrmap e29: y(active, since 1.98694s) 2024-09-20T22:40:40.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-mon[28216]: [20/Sep/2024:22:40:39] ENGINE Bus STARTING 2024-09-20T22:40:40.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-mon[28216]: [20/Sep/2024:22:40:40] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:40:40.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-mon[28216]: [20/Sep/2024:22:40:40] ENGINE Error in HTTPServer.serve 2024-09-20T22:40:40.908 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2024-09-20T22:40:40.908 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:40:40.908 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:40:40.908 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:40:40.908 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._run(expiration_interval) 2024-09-20T22:40:40.908 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: s = self.context.wrap_socket( 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.sslsocket_class._create( 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.do_handshake() 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._sslobj.do_handshake() 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-mon[28216]: [20/Sep/2024:22:40:40] ENGINE Serving on http://172.21.15.115:8765 2024-09-20T22:40:40.909 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:40 smithi115 ceph-mon[28216]: [20/Sep/2024:22:40:40] ENGINE Bus STARTED 2024-09-20T22:40:41.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:40 smithi153 ceph-mon[24571]: mgrmap e29: y(active, since 1.98694s) 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:40 smithi153 ceph-mon[24571]: [20/Sep/2024:22:40:39] ENGINE Bus STARTING 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:40 smithi153 ceph-mon[24571]: [20/Sep/2024:22:40:40] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:40 smithi153 ceph-mon[24571]: [20/Sep/2024:22:40:40] ENGINE Error in HTTPServer.serve 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: self._run(expiration_interval) 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: s = self.context.wrap_socket( 2024-09-20T22:40:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:40:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.sslsocket_class._create( 2024-09-20T22:40:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:40:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout: self.do_handshake() 2024-09-20T22:40:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:40:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout: self._sslobj.do_handshake() 2024-09-20T22:40:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:40:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:40 smithi153 ceph-mon[24571]: [20/Sep/2024:22:40:40] ENGINE Serving on http://172.21.15.115:8765 2024-09-20T22:40:41.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:40 smithi153 ceph-mon[24571]: [20/Sep/2024:22:40:40] ENGINE Bus STARTED 2024-09-20T22:40:41.503 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:41 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:41.052+0000 7f6575cf5040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:40:41.504 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:41 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:41.127+0000 7f6575cf5040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:40:41.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:41 smithi153 ceph-mon[24571]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:40:41.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:41 smithi115 ceph-mon[23720]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:40:41.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:41 smithi115 ceph-mon[28216]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:40:42.169 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:41 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:41.893+0000 7f6575cf5040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:40:42.169 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:42.021+0000 7f6575cf5040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:40:42.458 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:42.166+0000 7f6575cf5040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:40:42.458 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:42.232+0000 7f6575cf5040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:40:42.458 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:42.385+0000 7f6575cf5040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T22:40:42.458 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:42.455+0000 7f6575cf5040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:40:42.749 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:42.517+0000 7f6575cf5040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:40:42.749 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:42.680+0000 7f6575cf5040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:40:43.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:42 smithi153 ceph-mon[24571]: mgrmap e30: y(active, since 3s) 2024-09-20T22:40:43.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:42.757+0000 7f6575cf5040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:40:43.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:42.825+0000 7f6575cf5040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:40:43.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:42.910+0000 7f6575cf5040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:40:43.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:42 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:42.976+0000 7f6575cf5040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:40:43.086 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:42 smithi115 ceph-mon[23720]: mgrmap e30: y(active, since 3s) 2024-09-20T22:40:43.086 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:42 smithi115 ceph-mon[28216]: mgrmap e30: y(active, since 3s) 2024-09-20T22:40:43.506 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:43.036+0000 7f6575cf5040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:40:43.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:43.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:40:44.261 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:43.901+0000 7f6575cf5040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:40:44.261 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:43] ENGINE Bus STARTING 2024-09-20T22:40:44.262 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: CherryPy Checker: 2024-09-20T22:40:44.262 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: The Application mounted at '' has an empty config. 2024-09-20T22:40:44.262 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: 2024-09-20T22:40:44.262 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:44] ENGINE Serving on http://:::9283 2024-09-20T22:40:44.262 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: [20/Sep/2024:22:40:44] ENGINE Bus STARTED 2024-09-20T22:40:44.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-mon[24571]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:40:44.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-mon[24571]: Standby manager daemon x started 2024-09-20T22:40:44.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/2602386647' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:40:44.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/2602386647' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:40:44.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/2602386647' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:40:44.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/2602386647' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:40:44.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[23720]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:40:44.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[23720]: Standby manager daemon x started 2024-09-20T22:40:44.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/2602386647' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:40:44.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/2602386647' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:40:44.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/2602386647' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:40:44.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/2602386647' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:40:44.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[28216]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:40:44.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[28216]: Standby manager daemon x started 2024-09-20T22:40:44.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/2602386647' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:40:44.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/2602386647' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:40:44.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/2602386647' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:40:44.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/2602386647' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:40:44.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:44.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:44 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.028 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[45632]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:40:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-20T22:40:45.889 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: mgrmap e31: y(active, since 7s), standbys: x 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:40:45.890 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:40:45.891 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.891 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.891 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.891 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.891 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.891 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:40:45.891 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:45.891 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.891 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.891 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:45.891 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: mgrmap e31: y(active, since 7s), standbys: x 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.892 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.893 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.893 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:45.893 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:40:45.893 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:45 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:46.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:46.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:46.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:46.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:46.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:40:46.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:40:46.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:40:46.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:40:46.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:40:46.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: mgrmap e31: y(active, since 7s), standbys: x 2024-09-20T22:40:46.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:40:46.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:40:46.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:40:46.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:40:46.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:40:46.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:46.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:46.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:46.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:46.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:46.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:40:46.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:45 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:46.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:46 smithi115 ceph-mon[23720]: Reconfiguring iscsi.foo.smithi115.hwguxp (dependencies changed)... 2024-09-20T22:40:46.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:46 smithi115 ceph-mon[23720]: Reconfiguring daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:40:46.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:46 smithi115 ceph-mon[28216]: Reconfiguring iscsi.foo.smithi115.hwguxp (dependencies changed)... 2024-09-20T22:40:46.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:46 smithi115 ceph-mon[28216]: Reconfiguring daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:40:46.930 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:46 smithi153 ceph-mon[24571]: Reconfiguring iscsi.foo.smithi115.hwguxp (dependencies changed)... 2024-09-20T22:40:46.930 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:46 smithi153 ceph-mon[24571]: Reconfiguring daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:40:47.732 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:47 smithi115 ceph-mon[23720]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-20T22:40:47.732 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:47 smithi115 ceph-mon[28216]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-20T22:40:48.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:47 smithi153 ceph-mon[24571]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-20T22:40:49.979 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:49 smithi115 ceph-mon[23720]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-20T22:40:49.979 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:49 smithi115 ceph-mon[28216]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-20T22:40:50.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:49 smithi153 ceph-mon[24571]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-20T22:40:51.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:50 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:51.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:50 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:51.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:50 smithi153 ceph-mon[24571]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T22:40:51.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:50 smithi153 ceph-mon[24571]: Reconfiguring daemon prometheus.a on smithi153 2024-09-20T22:40:51.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:50 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3824466073' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:40:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:50 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:50 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:50 smithi115 ceph-mon[23720]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T22:40:51.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:50 smithi115 ceph-mon[23720]: Reconfiguring daemon prometheus.a on smithi153 2024-09-20T22:40:51.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:50 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/3824466073' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:40:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:50 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:50 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:50 smithi115 ceph-mon[28216]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T22:40:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:50 smithi115 ceph-mon[28216]: Reconfiguring daemon prometheus.a on smithi153 2024-09-20T22:40:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:50 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3824466073' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:40:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:51 smithi115 ceph-mon[23720]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T22:40:52.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:51 smithi115 ceph-mon[28216]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T22:40:52.228 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:51 smithi153 ceph-mon[24571]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T22:40:53.465 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 systemd[1]: Stopping Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:40:53.757 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:53.464Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T22:40:53.757 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:53.464Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-20T22:40:53.757 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:53.464Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-20T22:40:53.757 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:53.464Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T22:40:53.757 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:53.464Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-20T22:40:53.757 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:53.464Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T22:40:53.757 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:53.464Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-20T22:40:53.757 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:53.464Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-20T22:40:53.757 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:53.464Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-20T22:40:53.757 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:53.465Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T22:40:53.757 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:53.465Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-20T22:40:53.758 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[49322]: ts=2024-09-20T22:40:53.465Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-20T22:40:53.758 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 podman[52553]: 2024-09-20 22:40:53.474262133 +0000 UTC m=+0.101152847 container died dfa004a5353398d8a18af360e7da44e6cd97dd7dee95bc113d81864054a7abed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:40:54.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:53 smithi153 ceph-mon[24571]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:40:54.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 podman[52553]: 2024-09-20 22:40:53.876280977 +0000 UTC m=+0.503171689 container cleanup dfa004a5353398d8a18af360e7da44e6cd97dd7dee95bc113d81864054a7abed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:40:54.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:53 smithi153 bash[52553]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T22:40:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:53 smithi115 ceph-mon[23720]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:40:54.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:53 smithi115 ceph-mon[28216]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:40:54.477 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 podman[52567]: 2024-09-20 22:40:54.122935216 +0000 UTC m=+0.650175829 container remove dfa004a5353398d8a18af360e7da44e6cd97dd7dee95bc113d81864054a7abed (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:40:54.478 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-20T22:40:54.478 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 systemd[1]: Stopped Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:40:54.478 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@prometheus.a.service: Consumed 1.168s CPU time. 2024-09-20T22:40:54.478 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 systemd[1]: Starting Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:40:54.744 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 podman[52657]: 2024-09-20 22:40:54.380537956 +0000 UTC m=+0.017231781 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-20T22:40:54.744 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 podman[52657]: 2024-09-20 22:40:54.490448905 +0000 UTC m=+0.127142763 container create cb068a98c110f12c21687974a1203265ec172fb1fd15f6bb6aa78c18b0eec275 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:40:54.744 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 podman[52657]: 2024-09-20 22:40:54.742901002 +0000 UTC m=+0.379594822 container init cb068a98c110f12c21687974a1203265ec172fb1fd15f6bb6aa78c18b0eec275 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:40:54.776 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 podman[52657]: 2024-09-20 22:40:54.746041114 +0000 UTC m=+0.382734930 container start cb068a98c110f12c21687974a1203265ec172fb1fd15f6bb6aa78c18b0eec275 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:40:54.776 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.769Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-20T22:40:54.776 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.769Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-20T22:40:54.776 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.769Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi153 (none))" 2024-09-20T22:40:54.776 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.769Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T22:40:54.776 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.769Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T22:40:54.776 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.772Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T22:40:54.777 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.772Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-20T22:40:54.777 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.773Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T22:40:54.777 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.773Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T22:40:54.777 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.775Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T22:40:54.777 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.775Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.548µs 2024-09-20T22:40:54.777 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.775Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T22:40:55.015 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:55.015 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.782Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-20T22:40:55.015 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.793Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-20T22:40:55.015 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.798Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-20T22:40:55.015 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.798Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-20T22:40:55.015 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.798Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.784µs wal_replay_duration=23.631336ms wbl_replay_duration=130ns total_replay_duration=23.692905ms 2024-09-20T22:40:55.015 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.801Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T22:40:55.015 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.801Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-20T22:40:55.015 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.801Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T22:40:55.015 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.822Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.632828ms db_storage=821ns remote_storage=1.117µs web_handler=386ns query_engine=579ns scrape=579.755µs scrape_sd=108.685µs notify=16.039µs notify_sd=16.175µs rules=19.497539ms tracing=5.419µs 2024-09-20T22:40:55.015 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.822Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-20T22:40:55.016 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:40:54.822Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T22:40:55.016 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 bash[52657]: cb068a98c110f12c21687974a1203265ec172fb1fd15f6bb6aa78c18b0eec275 2024-09-20T22:40:55.016 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:40:54 smithi153 systemd[1]: Started Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:40:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:54 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:55.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:54 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:40:56.114 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:40:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:40:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:40:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:40:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: Adding iSCSI gateway http://:@172.21.15.115:5000 to Dashboard 2024-09-20T22:40:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:40:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:40:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi115"}]: dispatch 2024-09-20T22:40:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi115"}]: dispatch 2024-09-20T22:40:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:40:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:40:56.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:56.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:40:56.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:56.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:56.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:40:56.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:40:56.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:56.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: Adding iSCSI gateway http://:@172.21.15.115:5000 to Dashboard 2024-09-20T22:40:56.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:40:56.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:40:56.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi115"}]: dispatch 2024-09-20T22:40:56.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi115"}]: dispatch 2024-09-20T22:40:56.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:56.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:40:56.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:40:56.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:55 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:56.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:40:56.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:56.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:56.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:40:56.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-20T22:40:56.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:56.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: Adding iSCSI gateway http://:@172.21.15.115:5000 to Dashboard 2024-09-20T22:40:56.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:40:56.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-20T22:40:56.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi115"}]: dispatch 2024-09-20T22:40:56.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi115"}]: dispatch 2024-09-20T22:40:56.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:40:56.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:40:56.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:40:56.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:55 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:56.764 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:40:56.764 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (63s) 12s ago 5m 18.7M - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:40:56.764 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (37s) 13s ago 5m 43.3M - dad864ee21e9 568386e3ebfd 2024-09-20T22:40:56.764 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 starting - - - - 2024-09-20T22:40:56.764 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283 running (107s) 13s ago 10m 95.9M - 19.1.1-367-g9e2ac70b cc9251b744a3 e181bec15dad 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (32s) 12s ago 12m 473M - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (12m) 12s ago 12m 50.3M 2048M 17.2.0 e1d6a67b021e 0b3311e89569 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (10m) 13s ago 10m 45.2M 2048M 17.2.0 e1d6a67b021e 62b3afaddd26 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (10m) 12s ago 10m 44.5M 2048M 17.2.0 e1d6a67b021e 3ee70f11001d 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (72s) 12s ago 6m 7936k - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (42s) 13s ago 6m 7898k - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (10m) 12s ago 10m 52.5M 2736M 17.2.0 e1d6a67b021e 06f9f63b0da2 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (9m) 12s ago 9m 52.8M 2736M 17.2.0 e1d6a67b021e b68eba671ad6 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (9m) 12s ago 9m 47.5M 2736M 17.2.0 e1d6a67b021e a368d2632c3f 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (8m) 12s ago 8m 53.1M 2736M 17.2.0 e1d6a67b021e d4ef32545846 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (8m) 13s ago 8m 55.0M 2992M 17.2.0 e1d6a67b021e e9316b3d7317 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (8m) 13s ago 8m 49.5M 2992M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (7m) 13s ago 7m 48.5M 2992M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:40:56.765 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (7m) 13s ago 7m 51.4M 2992M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:40:56.766 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 starting - - - - 2024-09-20T22:40:56.766 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (5m) 12s ago 5m 73.2M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:40:56.766 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (5m) 13s ago 5m 73.6M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:40:56.766 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (4m) 12s ago 4m 72.7M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:40:56.766 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (4m) 13s ago 4m 70.0M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:40:56.958 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:56 smithi153 ceph-mon[24571]: from='client.25303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:56.958 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:56 smithi153 ceph-mon[24571]: Upgrade: Updating mgr.x 2024-09-20T22:40:56.958 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:56 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:40:56.958 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:56 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:40:56.958 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:56 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:56.958 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:56 smithi153 ceph-mon[24571]: Deploying daemon mgr.x on smithi153 2024-09-20T22:40:56.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:56 smithi153 ceph-mon[24571]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:57.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[23720]: from='client.25303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:57.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[23720]: Upgrade: Updating mgr.x 2024-09-20T22:40:57.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:40:57.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:40:57.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:57.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[23720]: Deploying daemon mgr.x on smithi153 2024-09-20T22:40:57.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[23720]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:57.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[28216]: from='client.25303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:57.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[28216]: Upgrade: Updating mgr.x 2024-09-20T22:40:57.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:40:57.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:40:57.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:57.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[28216]: Deploying daemon mgr.x on smithi153 2024-09-20T22:40:57.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:56 smithi115 ceph-mon[28216]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:57.163 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:40:57.163 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:40:57.163 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-09-20T22:40:57.163 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:40:57.163 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:40:57.163 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:40:57.165 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:40:57.165 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:40:57.165 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T22:40:57.165 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:40:57.165 INFO:teuthology.orchestra.run.smithi115.stdout: "mds": {}, 2024-09-20T22:40:57.166 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:40:57.166 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:40:57.166 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:40:57.166 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:40:57.166 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-09-20T22:40:57.166 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:40:57.166 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:40:57.166 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:40:57.492 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:40:57.493 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:40:57.493 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:40:57.493 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:40:57.493 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:40:57.493 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr" 2024-09-20T22:40:57.493 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:40:57.493 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "2/23 daemons upgraded", 2024-09-20T22:40:57.493 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading mgr daemons", 2024-09-20T22:40:57.493 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:40:57.493 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:40:57.888 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:40:58.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:57 smithi153 ceph-mon[24571]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:40:58.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:57 smithi153 ceph-mon[24571]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:58.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:57 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/581628559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:40:58.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:57 smithi153 ceph-mon[24571]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:58.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:57 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/402831406' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:40:58.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:57 smithi115 ceph-mon[23720]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:40:58.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:57 smithi115 ceph-mon[23720]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:58.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:57 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/581628559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:40:58.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:57 smithi115 ceph-mon[23720]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:58.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:57 smithi115 ceph-mon[23720]: from='client.? 172.21.15.115:0/402831406' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:40:58.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:57 smithi115 ceph-mon[28216]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:40:58.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:57 smithi115 ceph-mon[28216]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:58.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:57 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/581628559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:40:58.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:57 smithi115 ceph-mon[28216]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:58.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:57 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/402831406' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:40:58.302 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:58 smithi153 systemd[1]: Stopping Ceph mgr.x for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:40:58.571 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:58 smithi153 podman[52994]: 2024-09-20 22:40:58.300763142 +0000 UTC m=+0.118749831 container died e181bec15dad02e43f92269f1dfd2556c649c8ac9685f57d168714f8546c7865 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:40:58.837 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:58 smithi153 podman[52994]: 2024-09-20 22:40:58.643584555 +0000 UTC m=+0.461571241 container cleanup e181bec15dad02e43f92269f1dfd2556c649c8ac9685f57d168714f8546c7865 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:40:58.838 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:58 smithi153 bash[52994]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x 2024-09-20T22:40:59.269 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:58 smithi153 podman[53007]: 2024-09-20 22:40:58.836323433 +0000 UTC m=+0.532998434 container remove e181bec15dad02e43f92269f1dfd2556c649c8ac9685f57d168714f8546c7865 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-20T22:40:59.269 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:58 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-20T22:40:59.269 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:59 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-20T22:40:59.269 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:59 smithi153 systemd[1]: Stopped Ceph mgr.x for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:40:59.269 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:59 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.x.service: Consumed 18.858s CPU time. 2024-09-20T22:40:59.269 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:59 smithi153 systemd[1]: Starting Ceph mgr.x for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:40:59.525 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:59 smithi153 podman[53134]: 2024-09-20 22:40:59.381290871 +0000 UTC m=+0.016171383 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:40:59.525 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:59 smithi153 podman[53134]: 2024-09-20 22:40:59.524426158 +0000 UTC m=+0.159306665 container create 5de21dfaf787663865b0d98a8a5e3109583ec16e537441aa5df59dcd6ba933bc (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70) 2024-09-20T22:40:59.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:59 smithi153 podman[53134]: 2024-09-20 22:40:59.756500192 +0000 UTC m=+0.391380705 container init 5de21dfaf787663865b0d98a8a5e3109583ec16e537441aa5df59dcd6ba933bc (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70) 2024-09-20T22:40:59.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:59 smithi153 podman[53134]: 2024-09-20 22:40:59.759273226 +0000 UTC m=+0.394153738 container start 5de21dfaf787663865b0d98a8a5e3109583ec16e537441aa5df59dcd6ba933bc (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS) 2024-09-20T22:41:00.103 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:40:59 smithi153 ceph-mon[24571]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:41:00.104 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:59 smithi153 ceph-mgr[53153]: -- 172.21.15.153:0/919302528 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564ed4b510e0 con 0x564ed4b2f400 2024-09-20T22:41:00.104 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:59 smithi153 bash[53134]: 5de21dfaf787663865b0d98a8a5e3109583ec16e537441aa5df59dcd6ba933bc 2024-09-20T22:41:00.104 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:59 smithi153 systemd[1]: Started Ceph mgr.x for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:41:00.104 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:59 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:40:59.878+0000 7f862a730040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:41:00.104 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:40:59 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:40:59.943+0000 7f862a730040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:41:00.104 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:00 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:00.101+0000 7f862a730040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:41:00.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:40:59 smithi115 ceph-mon[23720]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:41:00.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:40:59 smithi115 ceph-mon[28216]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-20T22:41:00.491 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:00 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:00.171+0000 7f862a730040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:41:00.491 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:00 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:00.344+0000 7f862a730040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:41:00.491 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:00 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:00.489+0000 7f862a730040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:41:01.070 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:00 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:01.070 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:00 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:01.070 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:00 smithi153 ceph-mon[24571]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:01.070 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:00 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:00.815+0000 7f862a730040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:41:01.070 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:00 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:00.877+0000 7f862a730040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:41:01.218 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:00 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:01.218 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:00 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:01.218 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:00 smithi115 ceph-mon[23720]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:01.219 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:00 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:01.219 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:00 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:01.219 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:00 smithi115 ceph-mon[28216]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:01.384 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:01 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:01.067+0000 7f862a730040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:41:02.237 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:01 smithi115 ceph-mon[23720]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:02.238 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:01 smithi115 ceph-mon[28216]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:02.258 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:01 smithi153 ceph-mon[24571]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:02.258 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:02 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:02.011+0000 7f862a730040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:41:02.259 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:02 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:02.084+0000 7f862a730040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:41:03.093 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:02 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:02.825+0000 7f862a730040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:41:03.093 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:02 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:02.940+0000 7f862a730040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:41:03.376 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:03.091+0000 7f862a730040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:41:03.376 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:03.152+0000 7f862a730040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:41:03.376 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:03.306+0000 7f862a730040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T22:41:03.377 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:03.374+0000 7f862a730040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:41:03.677 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:41:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:41:03.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:41:03.677 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:03.434+0000 7f862a730040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:41:03.677 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:03.610+0000 7f862a730040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:41:03.677 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:03.675+0000 7f862a730040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:41:03.952 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:03.742+0000 7f862a730040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:41:03.952 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:03.831+0000 7f862a730040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:41:03.952 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:03.891+0000 7f862a730040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:41:04.267 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:03 smithi115 ceph-mon[23720]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:04.268 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:03 smithi115 ceph-mon[28216]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:04.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:03 smithi153 ceph-mon[24571]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:04.278 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:03.950+0000 7f862a730040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:41:05.057 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:04 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:04.804+0000 7f862a730040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:41:05.058 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:04 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: [20/Sep/2024:22:41:04] ENGINE Bus STARTING 2024-09-20T22:41:05.058 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:04 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: CherryPy Checker: 2024-09-20T22:41:05.058 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:04 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: The Application mounted at '' has an empty config. 2024-09-20T22:41:05.058 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:04 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:05.058 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:04 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: [20/Sep/2024:22:41:04] ENGINE Serving on http://:::9283 2024-09-20T22:41:05.058 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:04 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: [20/Sep/2024:22:41:04] ENGINE Bus STARTED 2024-09-20T22:41:05.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:05 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:05 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:05 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:05 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:05 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:05 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:05 smithi153 ceph-mon[24571]: Standby manager daemon x restarted 2024-09-20T22:41:05.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:05 smithi153 ceph-mon[24571]: Standby manager daemon x started 2024-09-20T22:41:05.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:05 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/2760963988' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:41:05.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:05 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/2760963988' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:41:05.363 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:05 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/2760963988' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:41:05.363 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:05 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/2760963988' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:41:05.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:41:05] "GET /metrics HTTP/1.1" 200 38181 "" "Prometheus/2.43.0" 2024-09-20T22:41:05.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[23720]: Standby manager daemon x restarted 2024-09-20T22:41:05.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[23720]: Standby manager daemon x started 2024-09-20T22:41:05.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/2760963988' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:41:05.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/2760963988' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:41:05.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/2760963988' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:41:05.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[23720]: from='mgr.? 172.21.15.153:0/2760963988' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:41:05.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:05.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[28216]: Standby manager daemon x restarted 2024-09-20T22:41:05.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[28216]: Standby manager daemon x started 2024-09-20T22:41:05.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/2760963988' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:41:05.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/2760963988' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:41:05.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/2760963988' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:41:05.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:05 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/2760963988' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:41:06.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:06 smithi115 ceph-mon[23720]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:06.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:06 smithi115 ceph-mon[23720]: mgrmap e32: y(active, since 28s), standbys: x 2024-09-20T22:41:06.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:06 smithi115 ceph-mon[28216]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:06.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:06 smithi115 ceph-mon[28216]: mgrmap e32: y(active, since 28s), standbys: x 2024-09-20T22:41:06.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:06 smithi153 ceph-mon[24571]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:06.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:06 smithi153 ceph-mon[24571]: mgrmap e32: y(active, since 28s), standbys: x 2024-09-20T22:41:08.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:08 smithi115 ceph-mon[23720]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:08.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:08 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:08.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:08 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:08.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:08 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:08.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:08 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:08.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:08 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:08.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:08 smithi115 ceph-mon[28216]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:08.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:08 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:08.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:08 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:08.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:08 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:08.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:08 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:08.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:08 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:08.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:08 smithi153 ceph-mon[24571]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:08 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:08 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:08 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:08 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:08 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:09.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:41:08 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:41:08.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:41:09.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:09.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:41:09.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[28216]: Upgrade: Setting container_image for all mgr 2024-09-20T22:41:09.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:09.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-20T22:41:09.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-20T22:41:09.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-20T22:41:09.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-20T22:41:09.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T22:41:09.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-20T22:41:09.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[28216]: Upgrade: It appears safe to stop mon.a 2024-09-20T22:41:09.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:41:09.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:09.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:41:09.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[23720]: Upgrade: Setting container_image for all mgr 2024-09-20T22:41:09.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:09.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-20T22:41:09.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-20T22:41:09.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-20T22:41:09.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-20T22:41:09.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T22:41:09.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-20T22:41:09.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[23720]: Upgrade: It appears safe to stop mon.a 2024-09-20T22:41:09.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:09 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:41:09.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:09 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:09.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:09 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:41:09.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:09 smithi153 ceph-mon[24571]: Upgrade: Setting container_image for all mgr 2024-09-20T22:41:09.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:09 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:09.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:09 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-20T22:41:09.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:09 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-20T22:41:09.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:09 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-20T22:41:09.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:09 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-20T22:41:09.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:09 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T22:41:09.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:09 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-20T22:41:09.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:09 smithi153 ceph-mon[24571]: Upgrade: It appears safe to stop mon.a 2024-09-20T22:41:09.531 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:09 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:41:10.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:10 smithi115 ceph-mon[23720]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:10.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:10 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:10.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:10 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:10.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:10 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:10.469 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:10 smithi115 ceph-mon[23720]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:10.469 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:10 smithi115 ceph-mon[28216]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:10.469 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:10 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:10.470 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:10 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:10.470 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:10 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:10.470 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:10 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:10.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:10 smithi153 ceph-mon[24571]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:10.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:10 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' 2024-09-20T22:41:10.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:10 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:10.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:10 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:10.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:10 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:11.467 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:11 smithi115 ceph-mon[23720]: Upgrade: Updating mon.a 2024-09-20T22:41:11.468 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:11 smithi115 ceph-mon[23720]: Deploying daemon mon.a on smithi115 2024-09-20T22:41:11.468 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:11 smithi115 ceph-mon[23720]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:11.468 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:11 smithi115 ceph-mon[28216]: Upgrade: Updating mon.a 2024-09-20T22:41:11.468 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:11 smithi115 ceph-mon[28216]: Deploying daemon mon.a on smithi115 2024-09-20T22:41:11.468 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:11 smithi115 ceph-mon[28216]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:11.735 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:11 smithi115 systemd[1]: Stopping Ceph mon.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:41:11.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:11 smithi153 ceph-mon[24571]: Upgrade: Updating mon.a 2024-09-20T22:41:11.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:11 smithi153 ceph-mon[24571]: Deploying daemon mon.a on smithi115 2024-09-20T22:41:11.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:11 smithi153 ceph-mon[24571]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:12.144 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:11 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:11] ENGINE Bus STOPPING 2024-09-20T22:41:12.144 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:11 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a[23716]: 2024-09-20T22:41:11.732+0000 7ffa7c9d2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:41:12.144 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:11 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a[23716]: 2024-09-20T22:41:11.732+0000 7ffa7c9d2700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-20T22:41:12.144 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:11 smithi115 podman[63942]: 2024-09-20 22:41:11.84590155 +0000 UTC m=+0.235652452 container died 0b3311e89569035e1e8559114a97e1f09352b0d04b427b43bb1ba6e8065f199a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, name=centos-stream, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_CLEAN=True, io.buildah.version=1.19.8, architecture=x86_64, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, version=8, RELEASE=HEAD, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container) 2024-09-20T22:41:12.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:12 smithi115 podman[63942]: 2024-09-20 22:41:12.241662552 +0000 UTC m=+0.631413453 container cleanup 0b3311e89569035e1e8559114a97e1f09352b0d04b427b43bb1ba6e8065f199a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, distribution-scope=public, maintainer=Guillaume Abrioux , version=8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, vcs-type=git, io.openshift.expose-services=, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-09-20T22:41:12.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:12 smithi115 bash[63942]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a 2024-09-20T22:41:12.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:41:12.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:12] ENGINE Bus STOPPED 2024-09-20T22:41:12.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:12] ENGINE Bus STARTING 2024-09-20T22:41:12.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:12] ENGINE Serving on http://:::9283 2024-09-20T22:41:12.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:12 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:12] ENGINE Bus STARTED 2024-09-20T22:41:12.884 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:12 smithi115 podman[63956]: 2024-09-20 22:41:12.5182339 +0000 UTC m=+0.664106886 container remove 0b3311e89569035e1e8559114a97e1f09352b0d04b427b43bb1ba6e8065f199a (image=quay.io/ceph/ceph:v17.2.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, GIT_CLEAN=True, RELEASE=HEAD, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, release=754, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, version=8, vcs-type=git, GIT_BRANCH=HEAD) 2024-09-20T22:41:12.884 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:12 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-20T22:41:12.884 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:12 smithi115 systemd[1]: Stopped Ceph mon.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:41:12.885 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:12 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.a.service: Consumed 13.582s CPU time. 2024-09-20T22:41:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:12 smithi115 systemd[1]: Starting Ceph mon.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:41:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 podman[64094]: 2024-09-20 22:41:12.995498845 +0000 UTC m=+0.015861558 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:41:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 podman[64094]: 2024-09-20 22:41:13.111156127 +0000 UTC m=+0.131518832 container create 29137eed6a0a64cd5d1ea780f5d3ca5c47919edf6c4795234f98ce6cec1e73ad (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-20T22:41:13.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 podman[64094]: 2024-09-20 22:41:13.322106259 +0000 UTC m=+0.342468966 container init 29137eed6a0a64cd5d1ea780f5d3ca5c47919edf6c4795234f98ce6cec1e73ad (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:41:13.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 podman[64094]: 2024-09-20 22:41:13.325713125 +0000 UTC m=+0.346075830 container start 29137eed6a0a64cd5d1ea780f5d3ca5c47919edf6c4795234f98ce6cec1e73ad (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-20T22:41:13.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable), process ceph-mon, pid 2 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: pidfile_write: ignore empty --pid-file 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: load: jerasure load: lrc 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: RocksDB version: 7.9.2 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Git sha 0 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Compile date 2024-09-19 18:20:48 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: DB SUMMARY 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: DB Session ID: O4NWK5Y7FFVBD6DSCB2P 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: CURRENT file: CURRENT 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1167 Bytes 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000033.sst 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000031.log size: 70053 ; 2024-09-20T22:41:13.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.error_if_exists: 0 2024-09-20T22:41:13.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.create_if_missing: 0 2024-09-20T22:41:13.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.paranoid_checks: 1 2024-09-20T22:41:13.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T22:41:13.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T22:41:13.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T22:41:13.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.env: 0x55c8ab75b940 2024-09-20T22:41:13.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T22:41:13.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.info_log: 0x55c8ac5d96e0 2024-09-20T22:41:13.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T22:41:13.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.statistics: (nil) 2024-09-20T22:41:13.657 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.use_fsync: 0 2024-09-20T22:41:13.657 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_log_file_size: 0 2024-09-20T22:41:13.657 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T22:41:13.657 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T22:41:13.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T22:41:13.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T22:41:13.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.allow_fallocate: 1 2024-09-20T22:41:13.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T22:41:13.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T22:41:13.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.use_direct_reads: 0 2024-09-20T22:41:13.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T22:41:13.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T22:41:13.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.db_log_dir: 2024-09-20T22:41:13.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.wal_dir: 2024-09-20T22:41:13.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T22:41:13.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T22:41:13.658 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T22:41:13.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T22:41:13.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T22:41:13.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T22:41:13.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T22:41:13.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T22:41:13.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.write_buffer_manager: 0x55c8ac5e88c0 2024-09-20T22:41:13.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T22:41:13.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T22:41:13.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T22:41:13.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T22:41:13.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.unordered_write: 0 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.row_cache: None 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.wal_filter: None 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.two_write_queues: 0 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T22:41:13.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.wal_compression: 0 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.atomic_flush: 0 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.log_readahead_size: 0 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_background_jobs: 2 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_background_compactions: -1 2024-09-20T22:41:13.661 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_subcompactions: 1 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_open_files: -1 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T22:41:13.662 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_background_flushes: -1 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Compression algorithms supported: 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: kZSTD supported: 0 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: kXpressCompression supported: 0 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: kBZip2Compression supported: 0 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: kLZ4Compression supported: 1 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: kZlibCompression supported: 1 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: kSnappyCompression supported: 1 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T22:41:13.663 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.merge_operator: 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compaction_filter: None 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c8ac5d9320) 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout: cache_index_and_filter_blocks: 1 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout: pin_top_level_index_and_filter: 1 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout: index_type: 0 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout: data_block_index_type: 0 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout: index_shortening: 1 2024-09-20T22:41:13.664 INFO:journalctl@ceph.mon.a.smithi115.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: checksum: 4 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: no_block_cache: 0 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache: 0x55c8ac5cd1f0 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache_name: BinnedLRUCache 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache_options: 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: capacity : 536870912 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: num_shard_bits : 4 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: strict_capacity_limit : 0 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: high_pri_pool_ratio: 0.000 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_cache_compressed: (nil) 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: persistent_cache: (nil) 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_size: 4096 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_size_deviation: 10 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_restart_interval: 16 2024-09-20T22:41:13.665 INFO:journalctl@ceph.mon.a.smithi115.stdout: index_block_restart_interval: 1 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: metadata_block_size: 4096 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: partition_filters: 0 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: use_delta_encoding: 1 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: filter_policy: bloomfilter 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: whole_key_filtering: 1 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: verify_compression: 0 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: read_amp_bytes_per_bit: 0 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: format_version: 5 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: enable_index_compression: 1 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: block_align: 0 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: max_auto_readahead_size: 262144 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: prepopulate_block_cache: 0 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: initial_auto_readahead_size: 8192 2024-09-20T22:41:13.666 INFO:journalctl@ceph.mon.a.smithi115.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compression: NoCompression 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.num_levels: 7 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T22:41:13.667 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T22:41:13.668 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T22:41:13.669 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T22:41:13.670 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.inplace_update_support: 0 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.bloom_locality: 0 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.max_successive_merges: 0 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T22:41:13.671 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.ttl: 2592000 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.enable_blob_files: false 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.min_blob_size: 0 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-09-20T22:41:13.672 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 13744, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 39618671-acfa-43c3-bd7e-cf98cbe798a0 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726872073353229, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726872073354188, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 70644, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 13745, "largest_seqno": 13806, "table_properties": {"data_size": 69296, "index_size": 182, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 133, "raw_key_size": 1121, "raw_average_key_size": 26, "raw_value_size": 68334, "raw_average_value_size": 1589, "num_data_blocks": 8, "num_entries": 43, "num_filter_entries": 43, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726872073, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "39618671-acfa-43c3-bd7e-cf98cbe798a0", "db_session_id": "O4NWK5Y7FFVBD6DSCB2P", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726872073354323, "job": 1, "event": "recovery_finished"} 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c8ac5fae00 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: DB pointer 0x55c8ac6f6000 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T22:41:13.673 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** DB Stats ** 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** Compaction Stats [default] ** 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: L0 1/0 68.99 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 103.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T22:41:13.674 INFO:journalctl@ceph.mon.a.smithi115.stdout: L6 1/0 9.59 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T22:41:13.675 INFO:journalctl@ceph.mon.a.smithi115.stdout: Sum 2/0 9.66 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 103.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T22:41:13.675 INFO:journalctl@ceph.mon.a.smithi115.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 103.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T22:41:13.675 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2024-09-20T22:41:13.675 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** Compaction Stats [default] ** 2024-09-20T22:41:13.675 INFO:journalctl@ceph.mon.a.smithi115.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T22:41:13.675 INFO:journalctl@ceph.mon.a.smithi115.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T22:41:13.675 INFO:journalctl@ceph.mon.a.smithi115.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 103.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T22:41:13.675 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2024-09-20T22:41:13.675 INFO:journalctl@ceph.mon.a.smithi115.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T22:41:13.675 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2024-09-20T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi115.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi115.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi115.stdout: Cumulative compaction: 0.00 GB write, 16.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi115.stdout: Interval compaction: 0.00 GB write, 16.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi115.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi115.stdout: Block cache BinnedLRUCache@0x55c8ac5cd1f0#2 capacity: 512.00 MB usage: 0.44 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-09-20T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi115.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.17 KB,3.27826e-05%) IndexBlock(1,0.27 KB,5.06639e-05%) Misc(1,0.00 KB,0%) 2024-09-20T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2024-09-20T22:41:13.676 INFO:journalctl@ceph.mon.a.smithi115.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: starting mon.a rank 0 at public addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] at bind addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: mon.a@-1(???) e3 preinit fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: mon.a@-1(???).mds e1 new map 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: mon.a@-1(???).mds e1 print_map 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout: e1 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout: legacy client fscid: -1 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout: 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout: No filesystems configured 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: mon.a@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T22:41:13.677 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T22:41:13.678 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T22:41:13.678 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 ceph-mon[64113]: mon.a@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-09-20T22:41:13.678 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 bash[64094]: 29137eed6a0a64cd5d1ea780f5d3ca5c47919edf6c4795234f98ce6cec1e73ad 2024-09-20T22:41:13.678 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:13 smithi115 systemd[1]: Started Ceph mon.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:41:13.778 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:41:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:41:13.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:41:14.943 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ignoring --setuser ceph since I am not root 2024-09-20T22:41:14.943 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ignoring --setgroup ceph since I am not root 2024-09-20T22:41:14.943 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mgr[58989]: -- 172.21.15.115:0/3996090602 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556af509b0e0 con 0x556af5079400 2024-09-20T22:41:14.943 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:14.940+0000 7f25ea171040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:41:14.943 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:14.943 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: mon.a calling monitor election 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: monmap epoch 3 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: last_changed 2024-09-20T22:30:16.783199+0000 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: created 2024-09-20T22:28:44.690924+0000 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: min_mon_release 17 (quincy) 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: election_strategy: 1 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: 0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: 1: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: fsmap 2024-09-20T22:41:14.944 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T22:41:14.945 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: mgrmap e32: y(active, since 37s), standbys: x 2024-09-20T22:41:14.945 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: overall HEALTH_OK 2024-09-20T22:41:14.945 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: from='mgr.25201 ' entity='' 2024-09-20T22:41:14.945 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[64113]: mgrmap e33: y(active, since 37s), standbys: x 2024-09-20T22:41:14.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:14.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:14.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:14.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:14.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: mon.a calling monitor election 2024-09-20T22:41:14.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-20T22:41:14.945 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: monmap epoch 3 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: last_changed 2024-09-20T22:30:16.783199+0000 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: created 2024-09-20T22:28:44.690924+0000 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: min_mon_release 17 (quincy) 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: election_strategy: 1 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: 0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: 1: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: fsmap 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: mgrmap e32: y(active, since 37s), standbys: x 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: overall HEALTH_OK 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: from='mgr.25201 ' entity='' 2024-09-20T22:41:14.946 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:14 smithi115 ceph-mon[28216]: mgrmap e33: y(active, since 37s), standbys: x 2024-09-20T22:41:15.142 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:15.143 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:15.143 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: from='mgr.25201 172.21.15.115:0/2839934347' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:15.143 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:15.143 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: mon.a calling monitor election 2024-09-20T22:41:15.143 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-20T22:41:15.143 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: monmap epoch 3 2024-09-20T22:41:15.143 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:15.143 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: last_changed 2024-09-20T22:30:16.783199+0000 2024-09-20T22:41:15.143 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: created 2024-09-20T22:28:44.690924+0000 2024-09-20T22:41:15.143 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: min_mon_release 17 (quincy) 2024-09-20T22:41:15.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: election_strategy: 1 2024-09-20T22:41:15.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: 0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2024-09-20T22:41:15.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: 1: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2024-09-20T22:41:15.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-09-20T22:41:15.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: fsmap 2024-09-20T22:41:15.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T22:41:15.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: mgrmap e32: y(active, since 37s), standbys: x 2024-09-20T22:41:15.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: overall HEALTH_OK 2024-09-20T22:41:15.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: from='mgr.25201 ' entity='' 2024-09-20T22:41:15.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mon[24571]: mgrmap e33: y(active, since 37s), standbys: x 2024-09-20T22:41:15.145 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: ignoring --setuser ceph since I am not root 2024-09-20T22:41:15.145 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: ignoring --setgroup ceph since I am not root 2024-09-20T22:41:15.145 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-mgr[53153]: -- 172.21.15.153:0/3926705200 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561bf0ad10e0 con 0x561bf0aaf000 2024-09-20T22:41:15.145 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:14.939+0000 7f46b123b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:41:15.145 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:14.995+0000 7f46b123b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:41:15.145 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:15 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:15.140+0000 7f46b123b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:41:15.215 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:14.999+0000 7f25ea171040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:41:15.215 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:15.146+0000 7f25ea171040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:41:15.215 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:15.212+0000 7f25ea171040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:41:15.491 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:15 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:15.206+0000 7f46b123b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:41:15.492 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:15 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:15.351+0000 7f46b123b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:41:15.500 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:15.358+0000 7f25ea171040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:41:16.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:15.498+0000 7f25ea171040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:41:16.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:15.814+0000 7f25ea171040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:41:16.155 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:15.874+0000 7f25ea171040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:41:16.155 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:16 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:16.058+0000 7f25ea171040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:41:16.278 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:15 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:15.489+0000 7f46b123b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:41:16.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:15 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:15.801+0000 7f46b123b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:41:16.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:15 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:15.860+0000 7f46b123b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:41:16.279 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:16 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:16.043+0000 7f46b123b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:41:17.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:16 smithi153 ceph-mon[24571]: mgrmap e34: y(active, since 38s), standbys: x 2024-09-20T22:41:17.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:16 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:16.968+0000 7f46b123b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:41:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:16 smithi115 ceph-mon[64113]: mgrmap e34: y(active, since 38s), standbys: x 2024-09-20T22:41:17.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:16 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:16.939+0000 7f25ea171040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:41:17.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:17 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:17.007+0000 7f25ea171040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:41:17.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:16 smithi115 ceph-mon[28216]: mgrmap e34: y(active, since 38s), standbys: x 2024-09-20T22:41:17.493 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:17.039+0000 7f46b123b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:41:17.976 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:17 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:17.714+0000 7f25ea171040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:41:17.977 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:17 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:17.832+0000 7f25ea171040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:41:18.028 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:17.770+0000 7f46b123b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:41:18.028 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:17.887+0000 7f46b123b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:41:18.253 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:17 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:17.974+0000 7f25ea171040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:41:18.253 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:18 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:18.035+0000 7f25ea171040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:41:18.253 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:18 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:18.183+0000 7f25ea171040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T22:41:18.309 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:18.031+0000 7f46b123b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:41:18.309 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:18.092+0000 7f46b123b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:41:18.309 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:18.240+0000 7f46b123b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T22:41:18.309 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:18.307+0000 7f46b123b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:41:18.536 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:18 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:18.250+0000 7f25ea171040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:41:18.536 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:18 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:18.310+0000 7f25ea171040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:41:18.536 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:18 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:18.468+0000 7f25ea171040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:41:18.536 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:18 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:18.533+0000 7f25ea171040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:41:18.593 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:18.367+0000 7f46b123b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:41:18.593 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:18.526+0000 7f46b123b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:41:18.593 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:18.591+0000 7f46b123b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:41:18.804 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:18 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:18.600+0000 7f25ea171040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:41:18.804 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:18 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:18.682+0000 7f25ea171040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:41:18.804 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:18 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:18.742+0000 7f25ea171040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:41:18.804 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:18 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:18.801+0000 7f25ea171040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:41:18.860 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:18.657+0000 7f46b123b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:41:18.860 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:18.740+0000 7f46b123b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:41:18.860 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:18.799+0000 7f46b123b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:41:18.860 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:18.858+0000 7f46b123b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:41:19.869 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:19.593+0000 7f25ea171040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:41:19.870 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:19] ENGINE Bus STARTING 2024-09-20T22:41:19.870 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: CherryPy Checker: 2024-09-20T22:41:19.870 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: The Application mounted at '' has an empty config. 2024-09-20T22:41:19.870 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:19.870 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:19] ENGINE Serving on http://:::9283 2024-09-20T22:41:19.870 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:19] ENGINE Bus STARTED 2024-09-20T22:41:19.870 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: Active manager daemon y restarted 2024-09-20T22:41:19.870 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: Activating manager daemon y 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: mgrmap e35: y(active, starting, since 0.00749351s), standbys: x 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:41:19.871 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:19.872 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:19.872 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:41:19.872 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:41:19.872 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:41:19.872 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[64113]: Manager daemon y is now available 2024-09-20T22:41:19.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: Active manager daemon y restarted 2024-09-20T22:41:19.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: Activating manager daemon y 2024-09-20T22:41:19.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:41:19.872 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: mgrmap e35: y(active, starting, since 0.00749351s), standbys: x 2024-09-20T22:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:19.873 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:19.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:41:19.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:41:19.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:41:19.874 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:19 smithi115 ceph-mon[28216]: Manager daemon y is now available 2024-09-20T22:41:20.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:19.659+0000 7f46b123b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:41:20.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: [20/Sep/2024:22:41:19] ENGINE Bus STARTING 2024-09-20T22:41:20.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: CherryPy Checker: 2024-09-20T22:41:20.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: The Application mounted at '' has an empty config. 2024-09-20T22:41:20.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: 2024-09-20T22:41:20.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: [20/Sep/2024:22:41:19] ENGINE Serving on http://:::9283 2024-09-20T22:41:20.029 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x[53149]: [20/Sep/2024:22:41:19] ENGINE Bus STARTED 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: Active manager daemon y restarted 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: Activating manager daemon y 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: mgrmap e35: y(active, starting, since 0.00749351s), standbys: x 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:41:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:41:20.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:41:20.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:20.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:20.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:41:20.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:41:20.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:41:20.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:19 smithi153 ceph-mon[24571]: Manager daemon y is now available 2024-09-20T22:41:20.903 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:20.609+0000 7f2588e7a640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-20T22:41:20.903 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[64113]: Standby manager daemon x restarted 2024-09-20T22:41:20.903 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[64113]: Standby manager daemon x started 2024-09-20T22:41:20.903 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[64113]: from='mgr.? 172.21.15.153:0/1168438173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:41:20.903 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[64113]: from='mgr.? 172.21.15.153:0/1168438173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:41:20.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[64113]: from='mgr.? 172.21.15.153:0/1168438173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:41:20.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[64113]: from='mgr.? 172.21.15.153:0/1168438173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:41:20.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:41:20.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:41:20.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:41:20.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:41:20.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:41:20.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[64113]: mgrmap e36: y(active, since 1.01282s), standbys: x 2024-09-20T22:41:20.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[28216]: Standby manager daemon x restarted 2024-09-20T22:41:20.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[28216]: Standby manager daemon x started 2024-09-20T22:41:20.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/1168438173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:41:20.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/1168438173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:41:20.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/1168438173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:41:20.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[28216]: from='mgr.? 172.21.15.153:0/1168438173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:41:20.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:41:20.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:41:20.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:41:20.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:41:20.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:41:20.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:20 smithi115 ceph-mon[28216]: mgrmap e36: y(active, since 1.01282s), standbys: x 2024-09-20T22:41:20.974 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:20 smithi153 ceph-mon[24571]: Standby manager daemon x restarted 2024-09-20T22:41:20.974 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:20 smithi153 ceph-mon[24571]: Standby manager daemon x started 2024-09-20T22:41:20.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:20 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/1168438173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:41:20.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:20 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/1168438173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:41:20.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:20 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/1168438173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:41:20.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:20 smithi153 ceph-mon[24571]: from='mgr.? 172.21.15.153:0/1168438173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:41:20.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:20 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:41:20.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:20 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:41:20.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:20 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:41:20.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:20 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:41:20.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:20 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:41:20.976 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:20 smithi153 ceph-mon[24571]: mgrmap e36: y(active, since 1.01282s), standbys: x 2024-09-20T22:41:21.623 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:21.333+0000 7f25469ae640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:22:41:21] ENGINE Error in HTTPServer.serve 2024-09-20T22:41:21.623 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: Traceback (most recent call last): 2024-09-20T22:41:21.623 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:41:21.623 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: self._connections.run(self.expiration_interval) 2024-09-20T22:41:21.623 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:41:21.623 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: self._run(expiration_interval) 2024-09-20T22:41:21.623 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:41:21.624 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:41:21.624 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:41:21.624 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:41:21.624 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:41:21.624 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: s = self.context.wrap_socket( 2024-09-20T22:41:21.624 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:41:21.624 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: return self.sslsocket_class._create( 2024-09-20T22:41:21.624 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:41:21.624 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: self.do_handshake() 2024-09-20T22:41:21.624 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:41:21.624 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: self._sslobj.do_handshake() 2024-09-20T22:41:21.624 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:41:21.625 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:21.722 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:21 smithi153 ceph-mon[24571]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:21.722 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:21 smithi153 ceph-mon[24571]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:21.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:21.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-mon[64113]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:21.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-mon[28216]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:21.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:21 smithi115 ceph-mon[28216]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:22.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[64113]: [20/Sep/2024:22:41:21] ENGINE Bus STARTING 2024-09-20T22:41:22.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[64113]: [20/Sep/2024:22:41:21] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:41:22.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[64113]: [20/Sep/2024:22:41:21] ENGINE Error in HTTPServer.serve 2024-09-20T22:41:22.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: Traceback (most recent call last): 2024-09-20T22:41:22.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:41:22.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:41:22.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:41:22.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: self._run(expiration_interval) 2024-09-20T22:41:22.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:41:22.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:41:22.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:41:22.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:41:22.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:41:22.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: s = self.context.wrap_socket( 2024-09-20T22:41:22.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:41:22.905 INFO:journalctl@ceph.mon.a.smithi115.stdout: return self.sslsocket_class._create( 2024-09-20T22:41:22.906 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:41:22.906 INFO:journalctl@ceph.mon.a.smithi115.stdout: self.do_handshake() 2024-09-20T22:41:22.906 INFO:journalctl@ceph.mon.a.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:41:22.906 INFO:journalctl@ceph.mon.a.smithi115.stdout: self._sslobj.do_handshake() 2024-09-20T22:41:22.906 INFO:journalctl@ceph.mon.a.smithi115.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:41:22.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[64113]: [20/Sep/2024:22:41:21] ENGINE Serving on http://172.21.15.115:8765 2024-09-20T22:41:22.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[64113]: [20/Sep/2024:22:41:21] ENGINE Bus STARTED 2024-09-20T22:41:22.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[64113]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:22.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[64113]: mgrmap e37: y(active, since 2s), standbys: x 2024-09-20T22:41:22.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[28216]: [20/Sep/2024:22:41:21] ENGINE Bus STARTING 2024-09-20T22:41:22.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[28216]: [20/Sep/2024:22:41:21] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:41:22.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[28216]: [20/Sep/2024:22:41:21] ENGINE Error in HTTPServer.serve 2024-09-20T22:41:22.907 INFO:journalctl@ceph.mon.c.smithi115.stdout: Traceback (most recent call last): 2024-09-20T22:41:22.907 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:41:22.907 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:41:22.907 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:41:22.907 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._run(expiration_interval) 2024-09-20T22:41:22.908 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:41:22.908 INFO:journalctl@ceph.mon.c.smithi115.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:41:22.908 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:41:22.908 INFO:journalctl@ceph.mon.c.smithi115.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:41:22.908 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:41:22.908 INFO:journalctl@ceph.mon.c.smithi115.stdout: s = self.context.wrap_socket( 2024-09-20T22:41:22.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:41:22.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: return self.sslsocket_class._create( 2024-09-20T22:41:22.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:41:22.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: self.do_handshake() 2024-09-20T22:41:22.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:41:22.909 INFO:journalctl@ceph.mon.c.smithi115.stdout: self._sslobj.do_handshake() 2024-09-20T22:41:22.910 INFO:journalctl@ceph.mon.c.smithi115.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:41:22.910 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[28216]: [20/Sep/2024:22:41:21] ENGINE Serving on http://172.21.15.115:8765 2024-09-20T22:41:22.910 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[28216]: [20/Sep/2024:22:41:21] ENGINE Bus STARTED 2024-09-20T22:41:22.910 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[28216]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:22.910 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:22 smithi115 ceph-mon[28216]: mgrmap e37: y(active, since 2s), standbys: x 2024-09-20T22:41:23.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:22 smithi153 ceph-mon[24571]: [20/Sep/2024:22:41:21] ENGINE Bus STARTING 2024-09-20T22:41:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:22 smithi153 ceph-mon[24571]: [20/Sep/2024:22:41:21] ENGINE Serving on https://172.21.15.115:7150 2024-09-20T22:41:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:22 smithi153 ceph-mon[24571]: [20/Sep/2024:22:41:21] ENGINE Error in HTTPServer.serve 2024-09-20T22:41:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-09-20T22:41:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:41:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:41:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:41:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: self._run(expiration_interval) 2024-09-20T22:41:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:41:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:41:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:41:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:41:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:41:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout: s = self.context.wrap_socket( 2024-09-20T22:41:23.030 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:41:23.030 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.sslsocket_class._create( 2024-09-20T22:41:23.030 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:41:23.030 INFO:journalctl@ceph.mon.b.smithi153.stdout: self.do_handshake() 2024-09-20T22:41:23.030 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:41:23.030 INFO:journalctl@ceph.mon.b.smithi153.stdout: self._sslobj.do_handshake() 2024-09-20T22:41:23.030 INFO:journalctl@ceph.mon.b.smithi153.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:41:23.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:22 smithi153 ceph-mon[24571]: [20/Sep/2024:22:41:21] ENGINE Serving on http://172.21.15.115:8765 2024-09-20T22:41:23.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:22 smithi153 ceph-mon[24571]: [20/Sep/2024:22:41:21] ENGINE Bus STARTED 2024-09-20T22:41:23.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:22 smithi153 ceph-mon[24571]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:23.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:22 smithi153 ceph-mon[24571]: mgrmap e37: y(active, since 2s), standbys: x 2024-09-20T22:41:23.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:41:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:41:23.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:41:24.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:24 smithi115 ceph-mon[64113]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:24.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:24 smithi115 ceph-mon[64113]: mgrmap e38: y(active, since 4s), standbys: x 2024-09-20T22:41:24.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:24 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:24.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:24 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:24.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:24 smithi115 ceph-mon[28216]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:24.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:24 smithi115 ceph-mon[28216]: mgrmap e38: y(active, since 4s), standbys: x 2024-09-20T22:41:24.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:24 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:24.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:24 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:25.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:24 smithi153 ceph-mon[24571]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:25.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:24 smithi153 ceph-mon[24571]: mgrmap e38: y(active, since 4s), standbys: x 2024-09-20T22:41:25.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:24 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:25.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:24 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:25.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:41:25] "GET /metrics HTTP/1.1" 200 35387 "" "Prometheus/2.43.0" 2024-09-20T22:41:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:25 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:25 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:25 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:25 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:25 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:25 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:25 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:25 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:25 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:25 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:25 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:25 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.205 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:25 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.205 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:25 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.205 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:25 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.206 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:25 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.206 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:25 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:26.206 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:25 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:26 smithi115 ceph-mon[64113]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:27.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:26 smithi115 ceph-mon[28216]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:27.272 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:26 smithi153 ceph-mon[24571]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:28.270 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:41:29.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:41:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:41:28.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:41:29.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:28 smithi153 ceph-mon[24571]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-20T22:41:29.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:28 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:29.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:28 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:29.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:28 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:29.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:28 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:29.051 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:41:29.051 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (95s) 4s ago 6m 18.7M - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:41:29.051 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (70s) 4s ago 6m 43.6M - dad864ee21e9 568386e3ebfd 2024-09-20T22:41:29.051 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (39s) 4s ago 5m 49.6M - 3.5 e1d6a67b021e e7840bcf555a 2024-09-20T22:41:29.051 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (29s) 4s ago 10m 437M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:41:29.051 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (64s) 4s ago 12m 473M - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:41:29.051 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (16s) 4s ago 12m 26.5M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:41:29.051 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (11m) 4s ago 11m 48.8M 2048M 17.2.0 e1d6a67b021e 62b3afaddd26 2024-09-20T22:41:29.051 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (11m) 4s ago 11m 48.6M 2048M 17.2.0 e1d6a67b021e 3ee70f11001d 2024-09-20T22:41:29.051 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (104s) 4s ago 7m 13.1M - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:41:29.051 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (75s) 4s ago 7m 15.7M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:41:29.051 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (10m) 4s ago 10m 53.0M 2736M 17.2.0 e1d6a67b021e 06f9f63b0da2 2024-09-20T22:41:29.052 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (10m) 4s ago 10m 53.3M 2736M 17.2.0 e1d6a67b021e b68eba671ad6 2024-09-20T22:41:29.052 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (9m) 4s ago 9m 47.9M 2736M 17.2.0 e1d6a67b021e a368d2632c3f 2024-09-20T22:41:29.052 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (9m) 4s ago 9m 53.8M 2736M 17.2.0 e1d6a67b021e d4ef32545846 2024-09-20T22:41:29.052 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (9m) 4s ago 9m 55.7M 2992M 17.2.0 e1d6a67b021e e9316b3d7317 2024-09-20T22:41:29.052 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (8m) 4s ago 8m 49.9M 2992M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:41:29.052 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (8m) 4s ago 8m 49.0M 2992M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:41:29.052 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (7m) 4s ago 7m 51.9M 2992M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:41:29.052 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (34s) 4s ago 7m 48.7M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:41:29.052 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (5m) 4s ago 5m 73.7M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:41:29.052 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (5m) 4s ago 5m 74.3M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:41:29.052 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (5m) 4s ago 5m 73.3M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:41:29.052 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (5m) 4s ago 5m 70.7M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:41:29.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:28 smithi115 ceph-mon[64113]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-20T22:41:29.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:28 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:29.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:28 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:29.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:28 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:29.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:28 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:29.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:28 smithi115 ceph-mon[28216]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-20T22:41:29.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:28 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:29.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:28 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:29.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:28 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:29.053 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:28 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:29.478 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:41:29.478 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:41:29.478 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-09-20T22:41:29.478 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T22:41:29.478 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:41:29.478 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:41:29.478 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:41:29.478 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:41:29.478 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:41:29.478 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T22:41:29.478 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:41:29.478 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:41:29.478 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:41:29.479 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:41:29.479 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:41:29.479 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-09-20T22:41:29.479 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:41:29.479 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:41:29.479 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:41:29.827 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:41:29.827 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:41:29.827 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:41:29.827 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:41:29.827 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:41:29.827 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr" 2024-09-20T22:41:29.827 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:41:29.827 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "3/23 daemons upgraded", 2024-09-20T22:41:29.828 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading mon daemons", 2024-09-20T22:41:29.828 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:41:29.828 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:41:30.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:41:30.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:41:30.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:41:30.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: from='client.25372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: from='client.25378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T22:41:30.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/3933845391' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: from='client.25372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: from='client.25378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.157 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.157 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.157 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:30.157 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:41:30.157 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T22:41:30.157 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-20T22:41:30.157 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:29 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3933845391' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:41:30.235 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:41:30.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: Updating smithi115:/etc/ceph/ceph.conf 2024-09-20T22:41:30.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: Updating smithi153:/etc/ceph/ceph.conf 2024-09-20T22:41:30.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:41:30.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:41:30.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: from='client.25372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:30.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:41:30.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:41:30.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: Updating smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:41:30.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: Updating smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:41:30.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: from='client.25378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:30.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:30.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:41:30.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T22:41:30.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-20T22:41:30.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:29 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/3933845391' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:41:30.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[64113]: Upgrade: It appears safe to stop mon.c 2024-09-20T22:41:30.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[64113]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:30.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[64113]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-20T22:41:30.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[64113]: from='client.25396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:30.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/2525377498' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:41:30.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:30.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:30.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:30.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[28216]: Upgrade: It appears safe to stop mon.c 2024-09-20T22:41:30.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[28216]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:30.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[28216]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-20T22:41:30.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[28216]: from='client.25396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:30.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[28216]: from='client.? 172.21.15.115:0/2525377498' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:41:30.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[28216]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:30.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:30.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:30.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:30 smithi115 ceph-mon[28216]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:30 smithi153 ceph-mon[24571]: Upgrade: It appears safe to stop mon.c 2024-09-20T22:41:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:30 smithi153 ceph-mon[24571]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:30 smithi153 ceph-mon[24571]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-20T22:41:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:30 smithi153 ceph-mon[24571]: from='client.25396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:30 smithi153 ceph-mon[24571]: from='client.? 172.21.15.115:0/2525377498' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:41:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:30 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:30 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:30 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:30 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:32.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:31 smithi115 ceph-mon[64113]: Upgrade: Updating mon.c 2024-09-20T22:41:32.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:31 smithi115 ceph-mon[64113]: Deploying daemon mon.c on smithi115 2024-09-20T22:41:32.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:31 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:31 smithi115 ceph-mon[28216]: Upgrade: Updating mon.c 2024-09-20T22:41:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:31 smithi115 ceph-mon[28216]: Deploying daemon mon.c on smithi115 2024-09-20T22:41:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:31 smithi115 ceph-mon[28216]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:31 smithi115 systemd[1]: Stopping Ceph mon.c for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:41:32.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:31 smithi153 ceph-mon[24571]: Upgrade: Updating mon.c 2024-09-20T22:41:32.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:31 smithi153 ceph-mon[24571]: Deploying daemon mon.c on smithi115 2024-09-20T22:41:32.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:31 smithi153 ceph-mon[24571]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:32.521 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:32 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:32] ENGINE Bus STOPPING 2024-09-20T22:41:32.521 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:32 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c[28212]: 2024-09-20T22:41:32.195+0000 7f991cc68700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:41:32.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:32 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c[28212]: 2024-09-20T22:41:32.195+0000 7f991cc68700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-20T22:41:32.522 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:32 smithi115 podman[66448]: 2024-09-20 22:41:32.226744353 +0000 UTC m=+0.134207902 container died 3ee70f11001dbfc90e8af167f4b812b82525b682e5f3ac7d8b976fe6cc9eda14 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c, io.openshift.expose-services=, RELEASE=HEAD, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, ceph=True, maintainer=Guillaume Abrioux , distribution-scope=public, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, release=754, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8) 2024-09-20T22:41:32.816 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:32 smithi115 podman[66448]: 2024-09-20 22:41:32.620284369 +0000 UTC m=+0.527747908 container cleanup 3ee70f11001dbfc90e8af167f4b812b82525b682e5f3ac7d8b976fe6cc9eda14 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, name=centos-stream, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, ceph=True, release=754, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, vcs-type=git, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-09-20T22:41:32.816 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:32 smithi115 bash[66448]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c 2024-09-20T22:41:32.816 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:32 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:41:32.816 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:32 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:32] ENGINE Bus STOPPED 2024-09-20T22:41:32.816 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:32 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:32] ENGINE Bus STARTING 2024-09-20T22:41:32.817 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:32 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:32] ENGINE Serving on http://:::9283 2024-09-20T22:41:32.817 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:32 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:32] ENGINE Bus STARTED 2024-09-20T22:41:33.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:32 smithi115 podman[66462]: 2024-09-20 22:41:32.900544079 +0000 UTC m=+0.670549670 container remove 3ee70f11001dbfc90e8af167f4b812b82525b682e5f3ac7d8b976fe6cc9eda14 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.openshift.tags=base centos centos-stream, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, maintainer=Guillaume Abrioux , RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD) 2024-09-20T22:41:33.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.c.service: Deactivated successfully. 2024-09-20T22:41:33.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 systemd[1]: Stopped Ceph mon.c for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:41:33.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.c.service: Consumed 7.761s CPU time. 2024-09-20T22:41:33.575 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 systemd[1]: Starting Ceph mon.c for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:41:33.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 podman[66603]: 2024-09-20 22:41:33.359199624 +0000 UTC m=+0.016019713 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:41:33.576 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 podman[66603]: 2024-09-20 22:41:33.470594472 +0000 UTC m=+0.127414561 container create 887b72a1971ef9bd73d6f669768a983207ba2f23841285044bc882cbdec3cad8 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-20T22:41:33.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:41:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:41:33.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:41:34.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 podman[66603]: 2024-09-20 22:41:33.718433163 +0000 UTC m=+0.375253253 container init 887b72a1971ef9bd73d6f669768a983207ba2f23841285044bc882cbdec3cad8 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-20T22:41:34.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 podman[66603]: 2024-09-20 22:41:33.721843402 +0000 UTC m=+0.378663493 container start 887b72a1971ef9bd73d6f669768a983207ba2f23841285044bc882cbdec3cad8 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-20T22:41:34.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T22:41:34.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable), process ceph-mon, pid 2 2024-09-20T22:41:34.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: pidfile_write: ignore empty --pid-file 2024-09-20T22:41:34.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: load: jerasure load: lrc 2024-09-20T22:41:34.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: RocksDB version: 7.9.2 2024-09-20T22:41:34.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Git sha 0 2024-09-20T22:41:34.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Compile date 2024-09-19 18:20:48 2024-09-20T22:41:34.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: DB SUMMARY 2024-09-20T22:41:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: DB Session ID: D3RJDD05768XBTZNQTRA 2024-09-20T22:41:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: CURRENT file: CURRENT 2024-09-20T22:41:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T22:41:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-09-20T22:41:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-09-20T22:41:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 3169879 ; 2024-09-20T22:41:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.error_if_exists: 0 2024-09-20T22:41:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.create_if_missing: 0 2024-09-20T22:41:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.paranoid_checks: 1 2024-09-20T22:41:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T22:41:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T22:41:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T22:41:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.env: 0x56132ff75940 2024-09-20T22:41:34.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T22:41:34.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.info_log: 0x561330a876e0 2024-09-20T22:41:34.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T22:41:34.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.statistics: (nil) 2024-09-20T22:41:34.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.use_fsync: 0 2024-09-20T22:41:34.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_log_file_size: 0 2024-09-20T22:41:34.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T22:41:34.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T22:41:34.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T22:41:34.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T22:41:34.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.allow_fallocate: 1 2024-09-20T22:41:34.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T22:41:34.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T22:41:34.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.use_direct_reads: 0 2024-09-20T22:41:34.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T22:41:34.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T22:41:34.158 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.db_log_dir: 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.wal_dir: 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.write_buffer_manager: 0x561330a968c0 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T22:41:34.159 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T22:41:34.160 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T22:41:34.160 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T22:41:34.160 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T22:41:34.160 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T22:41:34.160 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.unordered_write: 0 2024-09-20T22:41:34.160 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T22:41:34.160 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T22:41:34.160 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T22:41:34.160 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T22:41:34.160 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.row_cache: None 2024-09-20T22:41:34.160 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.wal_filter: None 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.two_write_queues: 0 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.wal_compression: 0 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.atomic_flush: 0 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.log_readahead_size: 0 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T22:41:34.161 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_background_jobs: 2 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_background_compactions: -1 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_subcompactions: 1 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T22:41:34.162 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_open_files: -1 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_background_flushes: -1 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Compression algorithms supported: 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: kZSTD supported: 0 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: kXpressCompression supported: 0 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: kBZip2Compression supported: 0 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: kLZ4Compression supported: 1 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: kZlibCompression supported: 1 2024-09-20T22:41:34.163 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: kSnappyCompression supported: 1 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.merge_operator: 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compaction_filter: None 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561330a87320) 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout: cache_index_and_filter_blocks: 1 2024-09-20T22:41:34.164 INFO:journalctl@ceph.mon.c.smithi115.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T22:41:34.165 INFO:journalctl@ceph.mon.c.smithi115.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T22:41:34.165 INFO:journalctl@ceph.mon.c.smithi115.stdout: pin_top_level_index_and_filter: 1 2024-09-20T22:41:34.165 INFO:journalctl@ceph.mon.c.smithi115.stdout: index_type: 0 2024-09-20T22:41:34.165 INFO:journalctl@ceph.mon.c.smithi115.stdout: data_block_index_type: 0 2024-09-20T22:41:34.165 INFO:journalctl@ceph.mon.c.smithi115.stdout: index_shortening: 1 2024-09-20T22:41:34.165 INFO:journalctl@ceph.mon.c.smithi115.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T22:41:34.165 INFO:journalctl@ceph.mon.c.smithi115.stdout: checksum: 4 2024-09-20T22:41:34.165 INFO:journalctl@ceph.mon.c.smithi115.stdout: no_block_cache: 0 2024-09-20T22:41:34.165 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_cache: 0x561330a7b1f0 2024-09-20T22:41:34.165 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_cache_name: BinnedLRUCache 2024-09-20T22:41:34.165 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_cache_options: 2024-09-20T22:41:34.165 INFO:journalctl@ceph.mon.c.smithi115.stdout: capacity : 536870912 2024-09-20T22:41:34.165 INFO:journalctl@ceph.mon.c.smithi115.stdout: num_shard_bits : 4 2024-09-20T22:41:34.166 INFO:journalctl@ceph.mon.c.smithi115.stdout: strict_capacity_limit : 0 2024-09-20T22:41:34.166 INFO:journalctl@ceph.mon.c.smithi115.stdout: high_pri_pool_ratio: 0.000 2024-09-20T22:41:34.166 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_cache_compressed: (nil) 2024-09-20T22:41:34.166 INFO:journalctl@ceph.mon.c.smithi115.stdout: persistent_cache: (nil) 2024-09-20T22:41:34.166 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_size: 4096 2024-09-20T22:41:34.166 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_size_deviation: 10 2024-09-20T22:41:34.166 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_restart_interval: 16 2024-09-20T22:41:34.166 INFO:journalctl@ceph.mon.c.smithi115.stdout: index_block_restart_interval: 1 2024-09-20T22:41:34.166 INFO:journalctl@ceph.mon.c.smithi115.stdout: metadata_block_size: 4096 2024-09-20T22:41:34.166 INFO:journalctl@ceph.mon.c.smithi115.stdout: partition_filters: 0 2024-09-20T22:41:34.166 INFO:journalctl@ceph.mon.c.smithi115.stdout: use_delta_encoding: 1 2024-09-20T22:41:34.166 INFO:journalctl@ceph.mon.c.smithi115.stdout: filter_policy: bloomfilter 2024-09-20T22:41:34.166 INFO:journalctl@ceph.mon.c.smithi115.stdout: whole_key_filtering: 1 2024-09-20T22:41:34.167 INFO:journalctl@ceph.mon.c.smithi115.stdout: verify_compression: 0 2024-09-20T22:41:34.167 INFO:journalctl@ceph.mon.c.smithi115.stdout: read_amp_bytes_per_bit: 0 2024-09-20T22:41:34.167 INFO:journalctl@ceph.mon.c.smithi115.stdout: format_version: 5 2024-09-20T22:41:34.167 INFO:journalctl@ceph.mon.c.smithi115.stdout: enable_index_compression: 1 2024-09-20T22:41:34.167 INFO:journalctl@ceph.mon.c.smithi115.stdout: block_align: 0 2024-09-20T22:41:34.167 INFO:journalctl@ceph.mon.c.smithi115.stdout: max_auto_readahead_size: 262144 2024-09-20T22:41:34.167 INFO:journalctl@ceph.mon.c.smithi115.stdout: prepopulate_block_cache: 0 2024-09-20T22:41:34.167 INFO:journalctl@ceph.mon.c.smithi115.stdout: initial_auto_readahead_size: 8192 2024-09-20T22:41:34.167 INFO:journalctl@ceph.mon.c.smithi115.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T22:41:34.167 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T22:41:34.167 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T22:41:34.167 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compression: NoCompression 2024-09-20T22:41:34.167 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.num_levels: 7 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T22:41:34.168 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T22:41:34.169 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T22:41:34.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T22:41:34.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T22:41:34.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T22:41:34.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T22:41:34.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T22:41:34.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T22:41:34.170 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T22:41:34.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T22:41:34.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T22:41:34.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T22:41:34.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T22:41:34.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T22:41:34.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T22:41:34.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T22:41:34.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T22:41:34.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T22:41:34.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T22:41:34.171 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.inplace_update_support: 0 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T22:41:34.172 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T22:41:34.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T22:41:34.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T22:41:34.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.bloom_locality: 0 2024-09-20T22:41:34.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.max_successive_merges: 0 2024-09-20T22:41:34.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T22:41:34.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T22:41:34.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T22:41:34.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T22:41:34.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.ttl: 2592000 2024-09-20T22:41:34.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T22:41:34.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T22:41:34.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T22:41:34.173 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.enable_blob_files: false 2024-09-20T22:41:34.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.min_blob_size: 0 2024-09-20T22:41:34.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T22:41:34.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T22:41:34.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T22:41:34.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T22:41:34.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T22:41:34.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T22:41:34.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T22:41:34.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T22:41:34.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-20T22:41:34.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T22:41:34.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14556, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T22:41:34.174 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-20T22:41:34.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 16f22020-7906-4137-90c9-7c8edf40bce6 2024-09-20T22:41:34.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726872093748860, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-20T22:41:34.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-20T22:41:34.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726872093756459, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 2036393, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14561, "largest_seqno": 15107, "table_properties": {"data_size": 2032872, "index_size": 1777, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 6693, "raw_average_key_size": 24, "raw_value_size": 2026840, "raw_average_value_size": 7424, "num_data_blocks": 81, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726872093, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "16f22020-7906-4137-90c9-7c8edf40bce6", "db_session_id": "D3RJDD05768XBTZNQTRA", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-20T22:41:34.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726872093756576, "job": 1, "event": "recovery_finished"} 2024-09-20T22:41:34.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-20T22:41:34.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T22:41:34.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T22:41:34.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561330aa8e00 2024-09-20T22:41:34.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: DB pointer 0x561330ba4000 2024-09-20T22:41:34.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T22:41:34.175 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T22:41:34.175 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** DB Stats ** 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** Compaction Stats [default] ** 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: L0 1/0 1.94 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 564.9 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: L6 1/0 9.59 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T22:41:34.176 INFO:journalctl@ceph.mon.c.smithi115.stdout: Sum 2/0 11.54 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 564.9 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 564.9 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** Compaction Stats [default] ** 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 564.9 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T22:41:34.177 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: Cumulative compaction: 0.00 GB write, 158.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: Interval compaction: 0.00 GB write, 158.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: Block cache BinnedLRUCache@0x561330a7b1f0#2 capacity: 512.00 MB usage: 2.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.78 KB,0.000149012%) IndexBlock(1,1.83 KB,0.000348687%) Misc(1,0.00 KB,0%) 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: starting mon.c rank 1 at public addrs [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] at bind addrs [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: mon.c@-1(???) e3 preinit fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: mon.c@-1(???).mds e1 new map 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: mon.c@-1(???).mds e1 print_map 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: e1 2024-09-20T22:41:34.178 INFO:journalctl@ceph.mon.c.smithi115.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T22:41:34.179 INFO:journalctl@ceph.mon.c.smithi115.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T22:41:34.179 INFO:journalctl@ceph.mon.c.smithi115.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T22:41:34.179 INFO:journalctl@ceph.mon.c.smithi115.stdout: legacy client fscid: -1 2024-09-20T22:41:34.179 INFO:journalctl@ceph.mon.c.smithi115.stdout: 2024-09-20T22:41:34.179 INFO:journalctl@ceph.mon.c.smithi115.stdout: No filesystems configured 2024-09-20T22:41:34.179 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: mon.c@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T22:41:34.179 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T22:41:34.179 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T22:41:34.179 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T22:41:34.179 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 ceph-mon[66621]: mon.c@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-09-20T22:41:34.179 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 bash[66603]: 887b72a1971ef9bd73d6f669768a983207ba2f23841285044bc882cbdec3cad8 2024-09-20T22:41:34.179 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:33 smithi115 systemd[1]: Started Ceph mon.c for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:41:34.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:34 smithi153 ceph-mon[24571]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T22:41:34.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:34 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:34.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:34 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:34.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:34 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:34.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:34 smithi115 ceph-mon[64113]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T22:41:34.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:34 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:34.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:34 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:34.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:34 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:34.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:34 smithi115 ceph-mon[66621]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-20T22:41:34.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:34 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:34.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:34 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:34.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:34 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:35.501 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:41:35] "GET /metrics HTTP/1.1" 200 38142 "" "Prometheus/2.43.0" 2024-09-20T22:41:35.502 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:41:35.502 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: mon.c calling monitor election 2024-09-20T22:41:35.502 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: mon.a calling monitor election 2024-09-20T22:41:35.502 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-20T22:41:35.502 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: monmap epoch 3 2024-09-20T22:41:35.502 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:35.502 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: last_changed 2024-09-20T22:30:16.783199+0000 2024-09-20T22:41:35.502 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: created 2024-09-20T22:28:44.690924+0000 2024-09-20T22:41:35.502 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: min_mon_release 17 (quincy) 2024-09-20T22:41:35.502 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: election_strategy: 1 2024-09-20T22:41:35.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: 0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2024-09-20T22:41:35.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: 1: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2024-09-20T22:41:35.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-09-20T22:41:35.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: fsmap 2024-09-20T22:41:35.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:41:35.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: mgrmap e38: y(active, since 14s), standbys: x 2024-09-20T22:41:35.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: overall HEALTH_OK 2024-09-20T22:41:35.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:35.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:35.503 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:41:35.503 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:41:35.503 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: mon.c calling monitor election 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: mon.a calling monitor election 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: monmap epoch 3 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: last_changed 2024-09-20T22:30:16.783199+0000 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: created 2024-09-20T22:28:44.690924+0000 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: min_mon_release 17 (quincy) 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: election_strategy: 1 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: 0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: 1: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: fsmap 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: mgrmap e38: y(active, since 14s), standbys: x 2024-09-20T22:41:35.504 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: overall HEALTH_OK 2024-09-20T22:41:35.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:35.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:35.505 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:35 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:41:35.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:41:35.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: mon.c calling monitor election 2024-09-20T22:41:35.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: mon.a calling monitor election 2024-09-20T22:41:35.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-20T22:41:35.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: monmap epoch 3 2024-09-20T22:41:35.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:35.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: last_changed 2024-09-20T22:30:16.783199+0000 2024-09-20T22:41:35.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: created 2024-09-20T22:28:44.690924+0000 2024-09-20T22:41:35.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: min_mon_release 17 (quincy) 2024-09-20T22:41:35.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: election_strategy: 1 2024-09-20T22:41:35.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: 0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2024-09-20T22:41:35.531 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: 1: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2024-09-20T22:41:35.531 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-09-20T22:41:35.531 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: fsmap 2024-09-20T22:41:35.531 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:41:35.531 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: mgrmap e38: y(active, since 14s), standbys: x 2024-09-20T22:41:35.531 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: overall HEALTH_OK 2024-09-20T22:41:35.532 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:35.532 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:35.532 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:35 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:41:36.502 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:36 smithi115 ceph-mon[64113]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:41:36.502 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:36 smithi115 ceph-mon[66621]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:41:36.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:36 smithi153 ceph-mon[24571]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:41:39.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:38 smithi153 ceph-mon[24571]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:41:39.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:41:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:41:38.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:41:39.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:38 smithi115 ceph-mon[64113]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:41:39.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:38 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:38.767+0000 7f2588e7a640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-20T22:41:39.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:38 smithi115 ceph-mon[66621]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-20T22:41:40.262 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:39 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:40.262 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:39 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:40.262 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:39 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:40.262 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:39 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:40.262 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:39 smithi115 ceph-mon[64113]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:40.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:39 smithi115 ceph-mon[66621]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:40.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:39 smithi115 ceph-mon[66621]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:40.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:39 smithi115 ceph-mon[66621]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:40.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:39 smithi115 ceph-mon[66621]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:40.262 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:39 smithi115 ceph-mon[66621]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:40.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:39 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:39 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:39 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:39 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:39 smithi153 ceph-mon[24571]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:41.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:41 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:41.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:41 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:42.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:41 smithi153 ceph-mon[24571]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:42.732 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:42 smithi115 ceph-mon[64113]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:42.732 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:42 smithi115 ceph-mon[66621]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:43.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:42 smithi153 ceph-mon[24571]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:43.535 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:41:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:41:43.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:41:43.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:43 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:43.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:43 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:43.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:43 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:43.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:43 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:43.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:43 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:43.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:43 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:43.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:43 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:41:43.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:43 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T22:41:43.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:43 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-20T22:41:44.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:44.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:44.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:44.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:44.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:41:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T22:41:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-20T22:41:44.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[66621]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:44.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[66621]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:44.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:44.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:44.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[66621]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:44.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:44.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:41:44.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T22:41:44.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:43 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-20T22:41:44.841 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:44 smithi153 ceph-mon[24571]: Upgrade: It appears safe to stop mon.b 2024-09-20T22:41:44.841 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:44 smithi153 ceph-mon[24571]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:44.841 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:44 smithi153 ceph-mon[24571]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:44.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:44 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:44.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:44 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:44.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:44 smithi153 ceph-mon[24571]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:44 smithi115 ceph-mon[64113]: Upgrade: It appears safe to stop mon.b 2024-09-20T22:41:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:44 smithi115 ceph-mon[64113]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:44 smithi115 ceph-mon[64113]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:44 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:45.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:44 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:45.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:44 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:44 smithi115 ceph-mon[66621]: Upgrade: It appears safe to stop mon.b 2024-09-20T22:41:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:44 smithi115 ceph-mon[66621]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:44 smithi115 ceph-mon[66621]: from='mgr.25345 ' entity='mgr.y' 2024-09-20T22:41:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:44 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:44 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:44 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:45.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:45 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:41:45] "GET /metrics HTTP/1.1" 200 38142 "" "Prometheus/2.43.0" 2024-09-20T22:41:45.867 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:45 smithi153 ceph-mon[24571]: Upgrade: Updating mon.b 2024-09-20T22:41:45.867 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:45 smithi153 ceph-mon[24571]: Deploying daemon mon.b on smithi153 2024-09-20T22:41:46.135 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:45 smithi153 systemd[1]: Stopping Ceph mon.b for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:41:46.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:45 smithi115 ceph-mon[64113]: Upgrade: Updating mon.b 2024-09-20T22:41:46.151 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:45 smithi115 ceph-mon[64113]: Deploying daemon mon.b on smithi153 2024-09-20T22:41:46.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:45 smithi115 ceph-mon[66621]: Upgrade: Updating mon.b 2024-09-20T22:41:46.151 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:45 smithi115 ceph-mon[66621]: Deploying daemon mon.b on smithi153 2024-09-20T22:41:46.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:46 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:46] ENGINE Bus STOPPING 2024-09-20T22:41:46.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:46 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:41:46.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:46 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:46] ENGINE Bus STOPPED 2024-09-20T22:41:46.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:46 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:46] ENGINE Bus STARTING 2024-09-20T22:41:46.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:46 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:46] ENGINE Serving on http://:::9283 2024-09-20T22:41:46.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:46 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:41:46] ENGINE Bus STARTED 2024-09-20T22:41:46.434 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:46 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b[24554]: 2024-09-20T22:41:46.132+0000 7f22e8ee3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:41:46.434 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:46 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b[24554]: 2024-09-20T22:41:46.132+0000 7f22e8ee3700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-20T22:41:46.435 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:46 smithi153 podman[56422]: 2024-09-20 22:41:46.153617926 +0000 UTC m=+0.143824914 container died 62b3afaddd2673d388680c2ee1c24cf55f746373873fbd005cf106f0ba40f2e3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, version=8, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-type=git, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux ) 2024-09-20T22:41:46.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:46 smithi153 podman[56422]: 2024-09-20 22:41:46.549251345 +0000 UTC m=+0.539458338 container cleanup 62b3afaddd2673d388680c2ee1c24cf55f746373873fbd005cf106f0ba40f2e3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, RELEASE=HEAD, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, release=754) 2024-09-20T22:41:46.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:46 smithi153 bash[56422]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b 2024-09-20T22:41:47.175 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:46 smithi153 podman[56433]: 2024-09-20 22:41:46.809158025 +0000 UTC m=+0.652241062 container remove 62b3afaddd2673d388680c2ee1c24cf55f746373873fbd005cf106f0ba40f2e3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b, version=8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.buildah.version=1.19.8, release=754, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_BRANCH=HEAD, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0) 2024-09-20T22:41:47.175 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:46 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.b.service: Deactivated successfully. 2024-09-20T22:41:47.175 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:46 smithi153 systemd[1]: Stopped Ceph mon.b for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:41:47.175 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:46 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.b.service: Consumed 9.189s CPU time. 2024-09-20T22:41:47.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 systemd[1]: Starting Ceph mon.b for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:41:47.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 podman[56565]: 2024-09-20 22:41:47.293641522 +0000 UTC m=+0.016429540 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:41:47.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 podman[56565]: 2024-09-20 22:41:47.40169125 +0000 UTC m=+0.124479266 container create f4dde7cfd242a82e7d2a316f40e280c08448a302b28725c952f838c583f5d490 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-09-20T22:41:47.783 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 podman[56565]: 2024-09-20 22:41:47.671196625 +0000 UTC m=+0.393984652 container init f4dde7cfd242a82e7d2a316f40e280c08448a302b28725c952f838c583f5d490 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:41:47.783 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 podman[56565]: 2024-09-20 22:41:47.674621767 +0000 UTC m=+0.397409775 container start f4dde7cfd242a82e7d2a316f40e280c08448a302b28725c952f838c583f5d490 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-20T22:41:47.783 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T22:41:47.783 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable), process ceph-mon, pid 2 2024-09-20T22:41:47.783 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: pidfile_write: ignore empty --pid-file 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: load: jerasure load: lrc 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: RocksDB version: 7.9.2 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Git sha 0 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Compile date 2024-09-19 18:20:48 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: DB SUMMARY 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: DB Session ID: NY86L8K46NI7XDV6A8PN 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: CURRENT file: CURRENT 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 3855127 ; 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.error_if_exists: 0 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.create_if_missing: 0 2024-09-20T22:41:47.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.paranoid_checks: 1 2024-09-20T22:41:47.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T22:41:47.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T22:41:47.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T22:41:47.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.env: 0x562a0d26e940 2024-09-20T22:41:47.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T22:41:47.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.info_log: 0x562a0eefd6e0 2024-09-20T22:41:47.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T22:41:47.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.statistics: (nil) 2024-09-20T22:41:47.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.use_fsync: 0 2024-09-20T22:41:47.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_log_file_size: 0 2024-09-20T22:41:47.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T22:41:47.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T22:41:47.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T22:41:47.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T22:41:47.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.allow_fallocate: 1 2024-09-20T22:41:47.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T22:41:47.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T22:41:47.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.use_direct_reads: 0 2024-09-20T22:41:47.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T22:41:47.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T22:41:47.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.db_log_dir: 2024-09-20T22:41:47.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.wal_dir: 2024-09-20T22:41:47.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T22:41:47.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T22:41:47.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T22:41:47.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T22:41:47.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T22:41:47.789 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T22:41:47.789 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T22:41:47.789 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T22:41:47.789 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.write_buffer_manager: 0x562a0ef0c8c0 2024-09-20T22:41:47.789 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T22:41:47.789 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T22:41:47.789 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T22:41:47.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T22:41:47.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T22:41:47.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T22:41:47.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T22:41:47.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T22:41:47.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.unordered_write: 0 2024-09-20T22:41:47.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T22:41:47.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T22:41:47.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T22:41:47.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T22:41:47.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.row_cache: None 2024-09-20T22:41:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.wal_filter: None 2024-09-20T22:41:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T22:41:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T22:41:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.two_write_queues: 0 2024-09-20T22:41:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T22:41:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.wal_compression: 0 2024-09-20T22:41:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.atomic_flush: 0 2024-09-20T22:41:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T22:41:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T22:41:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T22:41:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.log_readahead_size: 0 2024-09-20T22:41:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T22:41:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T22:41:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T22:41:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T22:41:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T22:41:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T22:41:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T22:41:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_background_jobs: 2 2024-09-20T22:41:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_background_compactions: -1 2024-09-20T22:41:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_subcompactions: 1 2024-09-20T22:41:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T22:41:47.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T22:41:47.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T22:41:47.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T22:41:47.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T22:41:47.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T22:41:47.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T22:41:47.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T22:41:47.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_open_files: -1 2024-09-20T22:41:47.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T22:41:47.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T22:41:47.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T22:41:47.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T22:41:47.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_background_flushes: -1 2024-09-20T22:41:47.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Compression algorithms supported: 2024-09-20T22:41:47.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: kZSTD supported: 0 2024-09-20T22:41:47.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: kXpressCompression supported: 0 2024-09-20T22:41:47.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: kBZip2Compression supported: 0 2024-09-20T22:41:47.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T22:41:47.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: kLZ4Compression supported: 1 2024-09-20T22:41:47.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: kZlibCompression supported: 1 2024-09-20T22:41:47.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T22:41:47.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: kSnappyCompression supported: 1 2024-09-20T22:41:47.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T22:41:47.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T22:41:47.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-09-20T22:41:47.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T22:41:47.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T22:41:47.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.merge_operator: 2024-09-20T22:41:47.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compaction_filter: None 2024-09-20T22:41:47.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T22:41:47.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T22:41:47.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T22:41:47.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T22:41:47.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562a0eefd320) 2024-09-20T22:41:47.797 INFO:journalctl@ceph.mon.b.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-09-20T22:41:47.797 INFO:journalctl@ceph.mon.b.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T22:41:47.797 INFO:journalctl@ceph.mon.b.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T22:41:47.797 INFO:journalctl@ceph.mon.b.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-09-20T22:41:47.797 INFO:journalctl@ceph.mon.b.smithi153.stdout: index_type: 0 2024-09-20T22:41:47.797 INFO:journalctl@ceph.mon.b.smithi153.stdout: data_block_index_type: 0 2024-09-20T22:41:47.797 INFO:journalctl@ceph.mon.b.smithi153.stdout: index_shortening: 1 2024-09-20T22:41:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T22:41:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout: checksum: 4 2024-09-20T22:41:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout: no_block_cache: 0 2024-09-20T22:41:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache: 0x562a0eef11f0 2024-09-20T22:41:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-09-20T22:41:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache_options: 2024-09-20T22:41:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout: capacity : 536870912 2024-09-20T22:41:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout: num_shard_bits : 4 2024-09-20T22:41:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout: strict_capacity_limit : 0 2024-09-20T22:41:47.799 INFO:journalctl@ceph.mon.b.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-09-20T22:41:47.799 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache_compressed: (nil) 2024-09-20T22:41:47.799 INFO:journalctl@ceph.mon.b.smithi153.stdout: persistent_cache: (nil) 2024-09-20T22:41:47.799 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_size: 4096 2024-09-20T22:41:47.799 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_size_deviation: 10 2024-09-20T22:41:47.799 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_restart_interval: 16 2024-09-20T22:41:47.799 INFO:journalctl@ceph.mon.b.smithi153.stdout: index_block_restart_interval: 1 2024-09-20T22:41:47.799 INFO:journalctl@ceph.mon.b.smithi153.stdout: metadata_block_size: 4096 2024-09-20T22:41:47.799 INFO:journalctl@ceph.mon.b.smithi153.stdout: partition_filters: 0 2024-09-20T22:41:47.799 INFO:journalctl@ceph.mon.b.smithi153.stdout: use_delta_encoding: 1 2024-09-20T22:41:47.800 INFO:journalctl@ceph.mon.b.smithi153.stdout: filter_policy: bloomfilter 2024-09-20T22:41:47.800 INFO:journalctl@ceph.mon.b.smithi153.stdout: whole_key_filtering: 1 2024-09-20T22:41:47.800 INFO:journalctl@ceph.mon.b.smithi153.stdout: verify_compression: 0 2024-09-20T22:41:47.800 INFO:journalctl@ceph.mon.b.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-09-20T22:41:47.800 INFO:journalctl@ceph.mon.b.smithi153.stdout: format_version: 5 2024-09-20T22:41:47.800 INFO:journalctl@ceph.mon.b.smithi153.stdout: enable_index_compression: 1 2024-09-20T22:41:47.800 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_align: 0 2024-09-20T22:41:47.800 INFO:journalctl@ceph.mon.b.smithi153.stdout: max_auto_readahead_size: 262144 2024-09-20T22:41:47.800 INFO:journalctl@ceph.mon.b.smithi153.stdout: prepopulate_block_cache: 0 2024-09-20T22:41:47.800 INFO:journalctl@ceph.mon.b.smithi153.stdout: initial_auto_readahead_size: 8192 2024-09-20T22:41:47.800 INFO:journalctl@ceph.mon.b.smithi153.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T22:41:47.801 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T22:41:47.801 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T22:41:47.801 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compression: NoCompression 2024-09-20T22:41:47.801 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T22:41:47.801 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T22:41:47.801 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T22:41:47.801 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.num_levels: 7 2024-09-20T22:41:47.801 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T22:41:47.801 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T22:41:47.801 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T22:41:47.801 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T22:41:47.802 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T22:41:47.802 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T22:41:47.802 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T22:41:47.802 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T22:41:47.802 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T22:41:47.802 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T22:41:47.802 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T22:41:47.802 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T22:41:47.802 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T22:41:47.802 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T22:41:47.803 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T22:41:47.803 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T22:41:47.803 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T22:41:47.803 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T22:41:47.803 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T22:41:47.803 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T22:41:47.803 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T22:41:47.803 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T22:41:47.803 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T22:41:47.803 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T22:41:47.803 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T22:41:47.804 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T22:41:47.804 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T22:41:47.804 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T22:41:47.804 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T22:41:47.804 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T22:41:47.804 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T22:41:47.804 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T22:41:47.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T22:41:47.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T22:41:47.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T22:41:47.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T22:41:47.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T22:41:47.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T22:41:47.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T22:41:47.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T22:41:47.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T22:41:47.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T22:41:47.806 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T22:41:47.806 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T22:41:47.806 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T22:41:47.806 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T22:41:47.806 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T22:41:47.806 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T22:41:47.806 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T22:41:47.806 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T22:41:47.806 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T22:41:47.806 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T22:41:47.807 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T22:41:47.807 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T22:41:47.807 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.inplace_update_support: 0 2024-09-20T22:41:47.807 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T22:41:47.807 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T22:41:47.807 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T22:41:47.807 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T22:41:47.807 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.bloom_locality: 0 2024-09-20T22:41:47.807 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.max_successive_merges: 0 2024-09-20T22:41:47.807 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T22:41:47.808 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T22:41:47.808 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T22:41:47.808 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T22:41:47.808 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.ttl: 2592000 2024-09-20T22:41:47.808 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T22:41:47.808 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T22:41:47.808 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T22:41:47.808 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.enable_blob_files: false 2024-09-20T22:41:47.808 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.min_blob_size: 0 2024-09-20T22:41:47.808 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T22:41:47.808 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T22:41:47.809 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T22:41:47.809 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T22:41:47.809 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T22:41:47.809 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T22:41:47.809 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T22:41:47.809 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T22:41:47.809 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-20T22:41:47.809 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T22:41:47.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14463, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T22:41:47.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-20T22:41:47.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e6ae92d6-1e9a-422a-9858-b5d81debf604 2024-09-20T22:41:47.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726872107702107, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-20T22:41:47.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-20T22:41:47.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726872107711058, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 2426770, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14468, "largest_seqno": 15301, "table_properties": {"data_size": 2422248, "index_size": 2518, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 965, "raw_key_size": 9220, "raw_average_key_size": 24, "raw_value_size": 2413967, "raw_average_value_size": 6302, "num_data_blocks": 114, "num_entries": 383, "num_filter_entries": 383, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726872107, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e6ae92d6-1e9a-422a-9858-b5d81debf604", "db_session_id": "NY86L8K46NI7XDV6A8PN", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-20T22:41:47.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726872107711195, "job": 1, "event": "recovery_finished"} 2024-09-20T22:41:47.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-20T22:41:47.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T22:41:47.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T22:41:47.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562a0ef1ee00 2024-09-20T22:41:47.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: DB pointer 0x562a0f01a000 2024-09-20T22:41:47.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T22:41:47.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T22:41:47.811 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** DB Stats ** 2024-09-20T22:41:47.811 INFO:journalctl@ceph.mon.b.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T22:41:47.811 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T22:41:47.811 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T22:41:47.811 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T22:41:47.812 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T22:41:47.812 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T22:41:47.812 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T22:41:47.812 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-09-20T22:41:47.812 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** Compaction Stats [default] ** 2024-09-20T22:41:47.812 INFO:journalctl@ceph.mon.b.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T22:41:47.812 INFO:journalctl@ceph.mon.b.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T22:41:47.812 INFO:journalctl@ceph.mon.b.smithi153.stdout: L0 1/0 2.31 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 598.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T22:41:47.812 INFO:journalctl@ceph.mon.b.smithi153.stdout: L6 1/0 9.60 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T22:41:47.812 INFO:journalctl@ceph.mon.b.smithi153.stdout: Sum 2/0 11.91 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 598.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T22:41:47.812 INFO:journalctl@ceph.mon.b.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 598.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T22:41:47.813 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-09-20T22:41:47.813 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** Compaction Stats [default] ** 2024-09-20T22:41:47.813 INFO:journalctl@ceph.mon.b.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T22:41:47.813 INFO:journalctl@ceph.mon.b.smithi153.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T22:41:47.813 INFO:journalctl@ceph.mon.b.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 598.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T22:41:47.813 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-09-20T22:41:47.813 INFO:journalctl@ceph.mon.b.smithi153.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T22:41:47.813 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-09-20T22:41:47.813 INFO:journalctl@ceph.mon.b.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T22:41:47.814 INFO:journalctl@ceph.mon.b.smithi153.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-09-20T22:41:47.814 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T22:41:47.814 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T22:41:47.814 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T22:41:47.814 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T22:41:47.814 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative compaction: 0.00 GB write, 173.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:41:47.814 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval compaction: 0.00 GB write, 173.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:41:47.814 INFO:journalctl@ceph.mon.b.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T22:41:47.814 INFO:journalctl@ceph.mon.b.smithi153.stdout: Block cache BinnedLRUCache@0x562a0eef11f0#2 capacity: 512.00 MB usage: 3.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-20T22:41:47.814 INFO:journalctl@ceph.mon.b.smithi153.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.03 KB,0.000196695%) IndexBlock(1,2.58 KB,0.000491738%) Misc(1,0.00 KB,0%) 2024-09-20T22:41:47.814 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-09-20T22:41:47.815 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T22:41:47.815 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: starting mon.b rank 2 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:47.815 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: mon.b@-1(???) e3 preinit fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:47.815 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: mon.b@-1(???).mds e1 new map 2024-09-20T22:41:47.815 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: mon.b@-1(???).mds e1 print_map 2024-09-20T22:41:47.815 INFO:journalctl@ceph.mon.b.smithi153.stdout: e1 2024-09-20T22:41:47.815 INFO:journalctl@ceph.mon.b.smithi153.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T22:41:47.815 INFO:journalctl@ceph.mon.b.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T22:41:47.815 INFO:journalctl@ceph.mon.b.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T22:41:47.815 INFO:journalctl@ceph.mon.b.smithi153.stdout: legacy client fscid: -1 2024-09-20T22:41:47.816 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-09-20T22:41:47.816 INFO:journalctl@ceph.mon.b.smithi153.stdout: No filesystems configured 2024-09-20T22:41:47.816 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: mon.b@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T22:41:47.816 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T22:41:47.816 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T22:41:47.816 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T22:41:47.816 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 ceph-mon[56592]: mon.b@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2024-09-20T22:41:47.816 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 bash[56565]: f4dde7cfd242a82e7d2a316f40e280c08448a302b28725c952f838c583f5d490 2024-09-20T22:41:47.816 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:47 smithi153 systemd[1]: Started Ceph mon.b for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:41:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:48 smithi153 ceph-mon[56592]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:48 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:48 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:48.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:48 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:48.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:48 smithi115 ceph-mon[64113]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:48.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:48 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:48.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:48 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:48.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:48 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:48.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:48 smithi115 ceph-mon[66621]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:48.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:48 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:48.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:48 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:48.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:48 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:48.981 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:41:48 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:41:48.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:41:49.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: fsmap 2024-09-20T22:41:49.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:41:49.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: mgrmap e38: y(active, since 28s), standbys: x 2024-09-20T22:41:49.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: overall HEALTH_OK 2024-09-20T22:41:49.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: mon.a calling monitor election 2024-09-20T22:41:49.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: mon.c calling monitor election 2024-09-20T22:41:49.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: mon.b calling monitor election 2024-09-20T22:41:49.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:49.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:49.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-20T22:41:49.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:49.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: monmap epoch 4 2024-09-20T22:41:49.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:49.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: last_changed 2024-09-20T22:41:48.189644+0000 2024-09-20T22:41:49.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: created 2024-09-20T22:28:44.690924+0000 2024-09-20T22:41:49.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: min_mon_release 19 (squid) 2024-09-20T22:41:49.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: election_strategy: 1 2024-09-20T22:41:49.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: 0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2024-09-20T22:41:49.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: 1: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2024-09-20T22:41:49.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-09-20T22:41:49.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: fsmap 2024-09-20T22:41:49.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:41:49.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: mgrmap e38: y(active, since 28s), standbys: x 2024-09-20T22:41:49.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: overall HEALTH_OK 2024-09-20T22:41:49.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:49.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:49.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: fsmap 2024-09-20T22:41:49.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:41:49.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: mgrmap e38: y(active, since 28s), standbys: x 2024-09-20T22:41:49.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: overall HEALTH_OK 2024-09-20T22:41:49.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: mon.a calling monitor election 2024-09-20T22:41:49.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: mon.c calling monitor election 2024-09-20T22:41:49.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: mon.b calling monitor election 2024-09-20T22:41:49.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:49.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:49.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-20T22:41:49.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:49.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: monmap epoch 4 2024-09-20T22:41:49.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:49.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: last_changed 2024-09-20T22:41:48.189644+0000 2024-09-20T22:41:49.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: created 2024-09-20T22:28:44.690924+0000 2024-09-20T22:41:49.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: min_mon_release 19 (squid) 2024-09-20T22:41:49.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: election_strategy: 1 2024-09-20T22:41:49.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: 0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2024-09-20T22:41:49.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: 1: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2024-09-20T22:41:49.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-09-20T22:41:49.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: fsmap 2024-09-20T22:41:49.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:41:49.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: mgrmap e38: y(active, since 28s), standbys: x 2024-09-20T22:41:49.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: overall HEALTH_OK 2024-09-20T22:41:49.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:49.406 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:49.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: fsmap 2024-09-20T22:41:49.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:41:49.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: mgrmap e38: y(active, since 28s), standbys: x 2024-09-20T22:41:49.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: overall HEALTH_OK 2024-09-20T22:41:49.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: mon.a calling monitor election 2024-09-20T22:41:49.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: mon.c calling monitor election 2024-09-20T22:41:49.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: mon.b calling monitor election 2024-09-20T22:41:49.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:41:49.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:41:49.407 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-20T22:41:49.408 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:41:49.408 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: monmap epoch 4 2024-09-20T22:41:49.408 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: fsid 9fe0f662-779f-11ef-baf5-efdc52797490 2024-09-20T22:41:49.408 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: last_changed 2024-09-20T22:41:48.189644+0000 2024-09-20T22:41:49.408 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: created 2024-09-20T22:28:44.690924+0000 2024-09-20T22:41:49.408 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: min_mon_release 19 (squid) 2024-09-20T22:41:49.408 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: election_strategy: 1 2024-09-20T22:41:49.409 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: 0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2024-09-20T22:41:49.409 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: 1: [v2:172.21.15.115:3301/0,v1:172.21.15.115:6790/0] mon.c 2024-09-20T22:41:49.409 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: 2: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-09-20T22:41:49.409 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: fsmap 2024-09-20T22:41:49.409 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:41:49.409 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: mgrmap e38: y(active, since 28s), standbys: x 2024-09-20T22:41:49.409 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: overall HEALTH_OK 2024-09-20T22:41:49.409 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:49.409 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:50.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:50 smithi153 ceph-mon[56592]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:50.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:50 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:41:50.653 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:50 smithi115 ceph-mon[64113]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:50.653 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:50 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:41:50.653 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:50 smithi115 ceph-mon[66621]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:50.653 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:50 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:41:51.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:51 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:51.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:51 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:51.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:51 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:41:52.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:52 smithi153 ceph-mon[56592]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:52.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:52 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:52.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:52 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:52.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:52 smithi115 ceph-mon[64113]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:52.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:52 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:52.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:52 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:52.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:52 smithi115 ceph-mon[66621]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:52.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:52 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:52.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:52 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:53.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:52 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:41:52.723+0000 7f2588e7a640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-20T22:41:53.710 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:41:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:41:53.504Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:41:54.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:53 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:54.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:53 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:53 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:53 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:54.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:53 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:54.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:53 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:55.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:54 smithi153 ceph-mon[56592]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:54 smithi115 ceph-mon[64113]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:55.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:54 smithi115 ceph-mon[66621]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:55.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:41:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:41:55] "GET /metrics HTTP/1.1" 200 38152 "" "Prometheus/2.43.0" 2024-09-20T22:41:57.004 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[64113]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:57.005 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:57.005 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:57.005 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:57.005 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:57.005 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:57.005 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:57.005 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:57.005 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:57.005 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[66621]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:57.005 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:57.005 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:57.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:57.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:57.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:57.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:57.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:57.006 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:56 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:57.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:56 smithi153 ceph-mon[56592]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:41:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:56 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:56 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:56 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:56 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:56 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:56 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:56 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:56 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:57.948 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[64113]: Reconfiguring mon.a (monmap changed)... 2024-09-20T22:41:57.948 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[64113]: Reconfiguring daemon mon.a on smithi115 2024-09-20T22:41:57.949 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:57.949 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:57.949 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:41:57.949 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:41:57.949 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:57.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[66621]: Reconfiguring mon.a (monmap changed)... 2024-09-20T22:41:57.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[66621]: Reconfiguring daemon mon.a on smithi115 2024-09-20T22:41:57.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:57.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:57.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:41:57.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:41:57.949 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:57 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:58.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:57 smithi153 ceph-mon[56592]: Reconfiguring mon.a (monmap changed)... 2024-09-20T22:41:58.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:57 smithi153 ceph-mon[56592]: Reconfiguring daemon mon.a on smithi115 2024-09-20T22:41:58.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:57 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:58.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:57 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:58.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:57 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:41:58.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:57 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:41:58.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:57 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:58.738 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:41:58 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:41:58.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:41:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:58 smithi153 ceph-mon[56592]: Reconfiguring mgr.y (monmap changed)... 2024-09-20T22:41:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:58 smithi153 ceph-mon[56592]: Reconfiguring daemon mgr.y on smithi115 2024-09-20T22:41:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:58 smithi153 ceph-mon[56592]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:58 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:58 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:58 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:58 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:58 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:59.086 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[64113]: Reconfiguring mgr.y (monmap changed)... 2024-09-20T22:41:59.086 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[64113]: Reconfiguring daemon mgr.y on smithi115 2024-09-20T22:41:59.086 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[64113]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:59.086 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:59.087 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:59.087 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:59.087 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:59.087 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:59.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[66621]: Reconfiguring mgr.y (monmap changed)... 2024-09-20T22:41:59.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[66621]: Reconfiguring daemon mgr.y on smithi115 2024-09-20T22:41:59.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[66621]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:41:59.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:59.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:41:59.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:41:59.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:41:59.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:58 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:00.017 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:59 smithi115 ceph-mon[64113]: Reconfiguring mon.c (monmap changed)... 2024-09-20T22:42:00.017 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:41:59 smithi115 ceph-mon[64113]: Reconfiguring daemon mon.c on smithi115 2024-09-20T22:42:00.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:59 smithi115 ceph-mon[66621]: Reconfiguring mon.c (monmap changed)... 2024-09-20T22:42:00.017 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:41:59 smithi115 ceph-mon[66621]: Reconfiguring daemon mon.c on smithi115 2024-09-20T22:42:00.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:59 smithi153 ceph-mon[56592]: Reconfiguring mon.c (monmap changed)... 2024-09-20T22:42:00.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:41:59 smithi153 ceph-mon[56592]: Reconfiguring daemon mon.c on smithi115 2024-09-20T22:42:00.600 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:42:01.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:00 smithi153 ceph-mon[56592]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:00 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:00 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:00 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T22:42:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:00 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:01.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:00 smithi115 ceph-mon[64113]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:01.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:00 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:01.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:00 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:01.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:00 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T22:42:01.032 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:00 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:01.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:00 smithi115 ceph-mon[66621]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:01.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:00 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:01.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:00 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:01.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:00 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T22:42:01.032 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:00 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:01.275 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:42:01.275 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2m) 22s ago 6m 19.1M - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:42:01.275 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (102s) 9s ago 6m 43.7M - dad864ee21e9 568386e3ebfd 2024-09-20T22:42:01.275 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (71s) 22s ago 5m 49.8M - 3.5 e1d6a67b021e e7840bcf555a 2024-09-20T22:42:01.276 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (61s) 9s ago 11m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:42:01.276 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (97s) 22s ago 13m 488M - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:42:01.276 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (48s) 22s ago 13m 37.5M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:42:01.276 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (13s) 9s ago 11m 20.6M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 f4dde7cfd242 2024-09-20T22:42:01.276 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (27s) 22s ago 11m 20.2M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 887b72a1971e 2024-09-20T22:42:01.276 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2m) 22s ago 7m 15.3M - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:42:01.277 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (107s) 9s ago 7m 15.1M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:42:01.277 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (11m) 22s ago 11m 53.6M 2736M 17.2.0 e1d6a67b021e 06f9f63b0da2 2024-09-20T22:42:01.277 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (10m) 22s ago 10m 53.5M 2736M 17.2.0 e1d6a67b021e b68eba671ad6 2024-09-20T22:42:01.277 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (10m) 22s ago 10m 48.1M 2736M 17.2.0 e1d6a67b021e a368d2632c3f 2024-09-20T22:42:01.277 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (9m) 22s ago 9m 53.8M 2736M 17.2.0 e1d6a67b021e d4ef32545846 2024-09-20T22:42:01.277 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (9m) 9s ago 9m 55.9M 2992M 17.2.0 e1d6a67b021e e9316b3d7317 2024-09-20T22:42:01.277 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (9m) 9s ago 9m 50.1M 2992M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:42:01.278 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (8m) 9s ago 8m 48.4M 2992M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:42:01.278 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (8m) 9s ago 8m 52.8M 2992M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:42:01.278 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (66s) 9s ago 7m 51.4M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:42:01.278 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (6m) 22s ago 6m 73.9M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:42:01.278 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (6m) 9s ago 6m 74.7M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:42:01.278 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (6m) 22s ago 5m 73.4M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:42:01.279 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (5m) 9s ago 5m 71.0M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:42:01.702 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:42:01.702 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:42:01.702 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:42:01.702 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:42:01.702 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:42:01.702 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:42:01.702 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:42:01.702 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:42:01.702 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T22:42:01.703 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:42:01.703 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:42:01.703 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:42:01.703 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:42:01.703 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:42:01.703 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-09-20T22:42:01.703 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 5 2024-09-20T22:42:01.703 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:42:01.703 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:42:01.963 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[64113]: Reconfiguring osd.0 (monmap changed)... 2024-09-20T22:42:01.963 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[64113]: Reconfiguring daemon osd.0 on smithi115 2024-09-20T22:42:01.963 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[64113]: from='client.44169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:01.964 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:01.964 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:01.964 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:01.964 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T22:42:01.964 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:01.964 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3841237021' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:01.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[66621]: Reconfiguring osd.0 (monmap changed)... 2024-09-20T22:42:01.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[66621]: Reconfiguring daemon osd.0 on smithi115 2024-09-20T22:42:01.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[66621]: from='client.44169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:01.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:01.964 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:01.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:01.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T22:42:01.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:01.965 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:01 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/3841237021' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:01 smithi153 ceph-mon[56592]: Reconfiguring osd.0 (monmap changed)... 2024-09-20T22:42:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:01 smithi153 ceph-mon[56592]: Reconfiguring daemon osd.0 on smithi115 2024-09-20T22:42:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:01 smithi153 ceph-mon[56592]: from='client.44169 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:01 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:01 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:01 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:01 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T22:42:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:01 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:01 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/3841237021' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:02.048 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:42:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:42:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:42:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:42:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:42:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr", 2024-09-20T22:42:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "mon" 2024-09-20T22:42:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:42:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "5/23 daemons upgraded", 2024-09-20T22:42:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading mon daemons", 2024-09-20T22:42:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:42:02.049 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:42:02.452 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:42:03.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:02 smithi153 ceph-mon[56592]: from='client.44175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:03.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:02 smithi153 ceph-mon[56592]: from='client.44181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:03.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:02 smithi153 ceph-mon[56592]: Reconfiguring osd.1 (monmap changed)... 2024-09-20T22:42:03.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:02 smithi153 ceph-mon[56592]: Reconfiguring daemon osd.1 on smithi115 2024-09-20T22:42:03.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:02 smithi153 ceph-mon[56592]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:03.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:02 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/3553881105' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:42:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:02 smithi115 ceph-mon[64113]: from='client.44175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:02 smithi115 ceph-mon[64113]: from='client.44181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:02 smithi115 ceph-mon[64113]: Reconfiguring osd.1 (monmap changed)... 2024-09-20T22:42:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:02 smithi115 ceph-mon[64113]: Reconfiguring daemon osd.1 on smithi115 2024-09-20T22:42:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:02 smithi115 ceph-mon[64113]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:02 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3553881105' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:42:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:02 smithi115 ceph-mon[66621]: from='client.44175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:02 smithi115 ceph-mon[66621]: from='client.44181 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:02 smithi115 ceph-mon[66621]: Reconfiguring osd.1 (monmap changed)... 2024-09-20T22:42:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:02 smithi115 ceph-mon[66621]: Reconfiguring daemon osd.1 on smithi115 2024-09-20T22:42:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:02 smithi115 ceph-mon[66621]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:03.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:02 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/3553881105' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:42:03.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:03 smithi153 ceph-mon[56592]: from='client.44193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:03.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:03 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:03.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:03 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:03.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:03 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T22:42:03.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:03 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:03.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:03.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:42:04.087 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:03 smithi115 ceph-mon[64113]: from='client.44193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:04.087 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:03 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:04.087 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:03 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:04.087 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:03 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T22:42:04.087 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:03 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:04.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:03 smithi115 ceph-mon[66621]: from='client.44193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:04.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:03 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:04.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:03 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:04.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:03 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T22:42:04.088 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:03 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:04.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[64113]: Reconfiguring osd.2 (monmap changed)... 2024-09-20T22:42:04.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[64113]: Reconfiguring daemon osd.2 on smithi115 2024-09-20T22:42:04.790 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[64113]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T22:42:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:04.791 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[66621]: Reconfiguring osd.2 (monmap changed)... 2024-09-20T22:42:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[66621]: Reconfiguring daemon osd.2 on smithi115 2024-09-20T22:42:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[66621]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:04.791 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T22:42:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:04.792 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:04 smithi153 ceph-mon[56592]: Reconfiguring osd.2 (monmap changed)... 2024-09-20T22:42:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:04 smithi153 ceph-mon[56592]: Reconfiguring daemon osd.2 on smithi115 2024-09-20T22:42:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:04 smithi153 ceph-mon[56592]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T22:42:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:05.560 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:42:05 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:42:05] "GET /metrics HTTP/1.1" 200 38217 "" "Prometheus/2.43.0" 2024-09-20T22:42:05.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:05 smithi115 ceph-mon[64113]: Reconfiguring osd.3 (monmap changed)... 2024-09-20T22:42:05.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:05 smithi115 ceph-mon[64113]: Reconfiguring daemon osd.3 on smithi115 2024-09-20T22:42:05.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:05 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:05.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:05 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:05.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:05 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:42:05.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:05 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:05.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:05 smithi115 ceph-mon[66621]: Reconfiguring osd.3 (monmap changed)... 2024-09-20T22:42:05.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:05 smithi115 ceph-mon[66621]: Reconfiguring daemon osd.3 on smithi115 2024-09-20T22:42:05.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:05 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:05.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:05 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:05.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:05 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:42:05.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:05 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:06.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:05 smithi153 ceph-mon[56592]: Reconfiguring osd.3 (monmap changed)... 2024-09-20T22:42:06.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:05 smithi153 ceph-mon[56592]: Reconfiguring daemon osd.3 on smithi115 2024-09-20T22:42:06.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:05 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:06.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:05 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:06.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:05 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:42:06.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:05 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:06.934 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:06 smithi115 ceph-mon[64113]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:06.934 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:06 smithi115 ceph-mon[64113]: Reconfiguring rgw.foo.smithi115.cotebr (monmap changed)... 2024-09-20T22:42:06.934 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:06 smithi115 ceph-mon[64113]: Reconfiguring daemon rgw.foo.smithi115.cotebr on smithi115 2024-09-20T22:42:06.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:06 smithi115 ceph-mon[66621]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:06.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:06 smithi115 ceph-mon[66621]: Reconfiguring rgw.foo.smithi115.cotebr (monmap changed)... 2024-09-20T22:42:06.935 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:06 smithi115 ceph-mon[66621]: Reconfiguring daemon rgw.foo.smithi115.cotebr on smithi115 2024-09-20T22:42:07.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:06 smithi153 ceph-mon[56592]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:07.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:06 smithi153 ceph-mon[56592]: Reconfiguring rgw.foo.smithi115.cotebr (monmap changed)... 2024-09-20T22:42:07.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:06 smithi153 ceph-mon[56592]: Reconfiguring daemon rgw.foo.smithi115.cotebr on smithi115 2024-09-20T22:42:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:07 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:07 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:07 smithi153 ceph-mon[56592]: Reconfiguring rgw.smpl.smithi115.pccgis (monmap changed)... 2024-09-20T22:42:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:07 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:42:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:07 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:07 smithi153 ceph-mon[56592]: Reconfiguring daemon rgw.smpl.smithi115.pccgis on smithi115 2024-09-20T22:42:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:07 smithi153 ceph-mon[56592]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[64113]: Reconfiguring rgw.smpl.smithi115.pccgis (monmap changed)... 2024-09-20T22:42:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:42:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[64113]: Reconfiguring daemon rgw.smpl.smithi115.pccgis on smithi115 2024-09-20T22:42:08.291 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[64113]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:08.291 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[66621]: Reconfiguring rgw.smpl.smithi115.pccgis (monmap changed)... 2024-09-20T22:42:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:42:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[66621]: Reconfiguring daemon rgw.smpl.smithi115.pccgis on smithi115 2024-09-20T22:42:08.292 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:07 smithi115 ceph-mon[66621]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:08.950 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:08 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:08.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:42:09.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:09.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:09.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[64113]: Reconfiguring mon.b (monmap changed)... 2024-09-20T22:42:09.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:42:09.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:42:09.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:09.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[64113]: Reconfiguring daemon mon.b on smithi153 2024-09-20T22:42:09.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:09.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:09.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[66621]: Reconfiguring mon.b (monmap changed)... 2024-09-20T22:42:09.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:42:09.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:42:09.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:09.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:09 smithi115 ceph-mon[66621]: Reconfiguring daemon mon.b on smithi153 2024-09-20T22:42:09.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:09 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:09.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:09 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:09.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:09 smithi153 ceph-mon[56592]: Reconfiguring mon.b (monmap changed)... 2024-09-20T22:42:09.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:09 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:42:09.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:09 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:42:09.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:09 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:09.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:09 smithi153 ceph-mon[56592]: Reconfiguring daemon mon.b on smithi153 2024-09-20T22:42:10.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[66621]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:10.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:10.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:10.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[66621]: Reconfiguring mgr.x (monmap changed)... 2024-09-20T22:42:10.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:42:10.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:42:10.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:10.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[66621]: Reconfiguring daemon mgr.x on smithi153 2024-09-20T22:42:10.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[64113]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:10.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:10.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:10.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[64113]: Reconfiguring mgr.x (monmap changed)... 2024-09-20T22:42:10.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:42:10.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:42:10.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:10.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:10 smithi115 ceph-mon[64113]: Reconfiguring daemon mgr.x on smithi153 2024-09-20T22:42:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:10 smithi153 ceph-mon[56592]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:10 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:10 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:10 smithi153 ceph-mon[56592]: Reconfiguring mgr.x (monmap changed)... 2024-09-20T22:42:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:10 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:42:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:10 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:42:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:10 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:10 smithi153 ceph-mon[56592]: Reconfiguring daemon mgr.x on smithi153 2024-09-20T22:42:11.861 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:11 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:11.861 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:11.861 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:11.861 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T22:42:11.861 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:11 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T22:42:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:12.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:11 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:12.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:12.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:12.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T22:42:12.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:12 smithi153 ceph-mon[56592]: Reconfiguring osd.4 (monmap changed)... 2024-09-20T22:42:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:12 smithi153 ceph-mon[56592]: Reconfiguring daemon osd.4 on smithi153 2024-09-20T22:42:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:12 smithi153 ceph-mon[56592]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:12 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:12 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:12 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T22:42:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:12 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[64113]: Reconfiguring osd.4 (monmap changed)... 2024-09-20T22:42:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[64113]: Reconfiguring daemon osd.4 on smithi153 2024-09-20T22:42:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[64113]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:13.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:13.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:13.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T22:42:13.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[66621]: Reconfiguring osd.4 (monmap changed)... 2024-09-20T22:42:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[66621]: Reconfiguring daemon osd.4 on smithi153 2024-09-20T22:42:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[66621]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T22:42:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:12 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:13.773 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:13.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:42:14.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:13 smithi153 ceph-mon[56592]: Reconfiguring osd.5 (monmap changed)... 2024-09-20T22:42:14.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:13 smithi153 ceph-mon[56592]: Reconfiguring daemon osd.5 on smithi153 2024-09-20T22:42:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:13 smithi115 ceph-mon[64113]: Reconfiguring osd.5 (monmap changed)... 2024-09-20T22:42:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:13 smithi115 ceph-mon[64113]: Reconfiguring daemon osd.5 on smithi153 2024-09-20T22:42:14.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:13 smithi115 ceph-mon[66621]: Reconfiguring osd.5 (monmap changed)... 2024-09-20T22:42:14.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:13 smithi115 ceph-mon[66621]: Reconfiguring daemon osd.5 on smithi153 2024-09-20T22:42:14.876 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:14 smithi153 ceph-mon[56592]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:14.876 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:14 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:14.876 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:14 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:14.876 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:14 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T22:42:14.876 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:14 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:15.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:14 smithi115 ceph-mon[64113]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:15.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:14 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:15.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:14 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:15.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:14 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T22:42:15.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:14 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:15.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:14 smithi115 ceph-mon[66621]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:15.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:14 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:15.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:14 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:15.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:14 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T22:42:15.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:14 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:15.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:42:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:42:15] "GET /metrics HTTP/1.1" 200 38217 "" "Prometheus/2.43.0" 2024-09-20T22:42:16.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:15 smithi153 ceph-mon[56592]: Reconfiguring osd.6 (monmap changed)... 2024-09-20T22:42:16.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:15 smithi153 ceph-mon[56592]: Reconfiguring daemon osd.6 on smithi153 2024-09-20T22:42:16.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:16.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:16.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T22:42:16.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:16.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:15 smithi115 ceph-mon[66621]: Reconfiguring osd.6 (monmap changed)... 2024-09-20T22:42:16.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:15 smithi115 ceph-mon[66621]: Reconfiguring daemon osd.6 on smithi153 2024-09-20T22:42:16.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:16.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:16.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T22:42:16.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:16.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:15 smithi115 ceph-mon[64113]: Reconfiguring osd.6 (monmap changed)... 2024-09-20T22:42:16.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:15 smithi115 ceph-mon[64113]: Reconfiguring daemon osd.6 on smithi153 2024-09-20T22:42:16.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:16.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:16.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T22:42:16.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:17.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:16 smithi153 ceph-mon[56592]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:17.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:16 smithi153 ceph-mon[56592]: Reconfiguring osd.7 (monmap changed)... 2024-09-20T22:42:17.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:16 smithi153 ceph-mon[56592]: Reconfiguring daemon osd.7 on smithi153 2024-09-20T22:42:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:16 smithi115 ceph-mon[64113]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:16 smithi115 ceph-mon[64113]: Reconfiguring osd.7 (monmap changed)... 2024-09-20T22:42:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:16 smithi115 ceph-mon[64113]: Reconfiguring daemon osd.7 on smithi153 2024-09-20T22:42:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:16 smithi115 ceph-mon[66621]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:16 smithi115 ceph-mon[66621]: Reconfiguring osd.7 (monmap changed)... 2024-09-20T22:42:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:16 smithi115 ceph-mon[66621]: Reconfiguring daemon osd.7 on smithi153 2024-09-20T22:42:18.531 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:18 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:18.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:18 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:18.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:18 smithi153 ceph-mon[56592]: Reconfiguring rgw.foo.smithi153.vetrnf (monmap changed)... 2024-09-20T22:42:18.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:18 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:42:18.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:18 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:18.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:18 smithi153 ceph-mon[56592]: Reconfiguring daemon rgw.foo.smithi153.vetrnf on smithi153 2024-09-20T22:42:18.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:18 smithi153 ceph-mon[56592]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:18.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:18.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:18.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[64113]: Reconfiguring rgw.foo.smithi153.vetrnf (monmap changed)... 2024-09-20T22:42:18.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:42:18.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:18.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[64113]: Reconfiguring daemon rgw.foo.smithi153.vetrnf on smithi153 2024-09-20T22:42:18.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[64113]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:18.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:18.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:18.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[66621]: Reconfiguring rgw.foo.smithi153.vetrnf (monmap changed)... 2024-09-20T22:42:18.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:42:18.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:18.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[66621]: Reconfiguring daemon rgw.foo.smithi153.vetrnf on smithi153 2024-09-20T22:42:18.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:18 smithi115 ceph-mon[66621]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:19.028 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:18.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:42:20.002 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.002 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.002 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:19 smithi153 ceph-mon[56592]: Reconfiguring rgw.smpl.smithi153.oznbtc (monmap changed)... 2024-09-20T22:42:20.003 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:42:20.003 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:20.003 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:19 smithi153 ceph-mon[56592]: Reconfiguring daemon rgw.smpl.smithi153.oznbtc on smithi153 2024-09-20T22:42:20.003 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:20.031 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.032 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.032 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[64113]: Reconfiguring rgw.smpl.smithi153.oznbtc (monmap changed)... 2024-09-20T22:42:20.032 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:42:20.032 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:20.032 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[64113]: Reconfiguring daemon rgw.smpl.smithi153.oznbtc on smithi153 2024-09-20T22:42:20.032 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:20.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[66621]: Reconfiguring rgw.smpl.smithi153.oznbtc (monmap changed)... 2024-09-20T22:42:20.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:42:20.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:20.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[66621]: Reconfiguring daemon rgw.smpl.smithi153.oznbtc on smithi153 2024-09-20T22:42:20.033 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:42:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:20.793 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-20T22:42:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-20T22:42:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-20T22:42:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-20T22:42:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-20T22:42:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-20T22:42:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.794 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T22:42:20.794 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:20.795 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:20.796 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T22:42:21.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:21.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:21.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:21.032 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:42:21.032 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:21.032 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:21.032 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:21.032 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-20T22:42:21.032 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-20T22:42:21.032 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-20T22:42:21.033 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-20T22:42:21.033 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-20T22:42:21.033 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-20T22:42:21.033 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:21.033 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:21.033 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T22:42:21.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[64113]: Upgrade: Setting container_image for all mon 2024-09-20T22:42:21.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[64113]: Upgrade: Setting container_image for all crash 2024-09-20T22:42:21.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T22:42:21.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[64113]: Upgrade: osd.0 is safe to restart 2024-09-20T22:42:21.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:21.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:21.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T22:42:21.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:21.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[66621]: Upgrade: Setting container_image for all mon 2024-09-20T22:42:21.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[66621]: Upgrade: Setting container_image for all crash 2024-09-20T22:42:21.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T22:42:21.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[66621]: Upgrade: osd.0 is safe to restart 2024-09-20T22:42:21.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:21.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:21.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T22:42:21.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:21 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:22.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:21 smithi153 ceph-mon[56592]: Upgrade: Setting container_image for all mon 2024-09-20T22:42:22.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:21 smithi153 ceph-mon[56592]: Upgrade: Setting container_image for all crash 2024-09-20T22:42:22.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:21 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T22:42:22.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:21 smithi153 ceph-mon[56592]: Upgrade: osd.0 is safe to restart 2024-09-20T22:42:22.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:21 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:22.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:21 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:22.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:21 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T22:42:22.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:21 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:22.811 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:22 smithi115 ceph-mon[64113]: Upgrade: Updating osd.0 2024-09-20T22:42:22.811 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:22 smithi115 ceph-mon[64113]: Deploying daemon osd.0 on smithi115 2024-09-20T22:42:22.811 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:22 smithi115 ceph-mon[64113]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:22.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:22 smithi115 ceph-mon[66621]: Upgrade: Updating osd.0 2024-09-20T22:42:22.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:22 smithi115 ceph-mon[66621]: Deploying daemon osd.0 on smithi115 2024-09-20T22:42:22.812 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:22 smithi115 ceph-mon[66621]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:23.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:22 smithi153 ceph-mon[56592]: Upgrade: Updating osd.0 2024-09-20T22:42:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:22 smithi153 ceph-mon[56592]: Deploying daemon osd.0 on smithi115 2024-09-20T22:42:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:22 smithi153 ceph-mon[56592]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:23.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:23.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:42:24.654 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:24 smithi115 systemd[1]: Stopping Ceph osd.0 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:42:25.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:24 smithi153 ceph-mon[56592]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:25.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:24 smithi153 ceph-mon[56592]: osd.0 marked itself down and dead 2024-09-20T22:42:25.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:24 smithi115 ceph-mon[64113]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:25.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:24 smithi115 ceph-mon[64113]: osd.0 marked itself down and dead 2024-09-20T22:42:25.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:24 smithi115 ceph-mon[66621]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:25.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:24 smithi115 ceph-mon[66621]: osd.0 marked itself down and dead 2024-09-20T22:42:25.156 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:24 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0[31048]: 2024-09-20T22:42:24.663+0000 7fbfb92a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:42:25.156 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:24 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0[31048]: 2024-09-20T22:42:24.663+0000 7fbfb92a0700 -1 osd.0 106 *** Got signal Terminated *** 2024-09-20T22:42:25.156 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:24 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0[31048]: 2024-09-20T22:42:24.663+0000 7fbfb92a0700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:42:25.449 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:42:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:42:25] "GET /metrics HTTP/1.1" 200 38219 "" "Prometheus/2.43.0" 2024-09-20T22:42:25.450 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:25 smithi115 podman[69382]: 2024-09-20 22:42:25.166461605 +0000 UTC m=+0.623185542 container died 06f9f63b0da2fd366b5296e81d8c96100eb9e06b07d842a02b39ddd77f88af6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., version=8, io.openshift.tags=base centos centos-stream, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, name=centos-stream, io.openshift.expose-services=, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0) 2024-09-20T22:42:25.738 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:25 smithi115 podman[69382]: 2024-09-20 22:42:25.546834932 +0000 UTC m=+1.003558858 container cleanup 06f9f63b0da2fd366b5296e81d8c96100eb9e06b07d842a02b39ddd77f88af6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, architecture=x86_64, RELEASE=HEAD, vcs-type=git, GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, release=754, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.openshift.expose-services=) 2024-09-20T22:42:25.738 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:25 smithi115 bash[69382]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0 2024-09-20T22:42:25.740 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:25 smithi115 ceph-mon[64113]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:42:25.740 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:25 smithi115 ceph-mon[66621]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:42:26.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:25 smithi153 ceph-mon[56592]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:42:26.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:25 smithi153 ceph-mon[56592]: osdmap e107: 8 total, 7 up, 8 in 2024-09-20T22:42:26.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:25 smithi153 ceph-mon[56592]: osdmap e108: 8 total, 7 up, 8 in 2024-09-20T22:42:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:25 smithi115 ceph-mon[64113]: osdmap e107: 8 total, 7 up, 8 in 2024-09-20T22:42:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:25 smithi115 ceph-mon[64113]: osdmap e108: 8 total, 7 up, 8 in 2024-09-20T22:42:26.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:25 smithi115 ceph-mon[66621]: osdmap e107: 8 total, 7 up, 8 in 2024-09-20T22:42:26.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:25 smithi115 ceph-mon[66621]: osdmap e108: 8 total, 7 up, 8 in 2024-09-20T22:42:26.155 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:25 smithi115 podman[69408]: 2024-09-20 22:42:25.848759102 +0000 UTC m=+0.679551428 container remove 06f9f63b0da2fd366b5296e81d8c96100eb9e06b07d842a02b39ddd77f88af6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, distribution-scope=public, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, name=centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.openshift.expose-services=, vcs-type=git, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.openshift.tags=base centos centos-stream) 2024-09-20T22:42:26.469 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:26 smithi115 podman[69497]: 2024-09-20 22:42:26.105064179 +0000 UTC m=+0.015929764 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:42:26.470 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:26 smithi115 podman[69497]: 2024-09-20 22:42:26.232883824 +0000 UTC m=+0.143749393 container create 3c6347d2a89122f3921d716a8bfb3a7c46a3c2387250313de367c2fc0d55a513 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:42:26.470 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:26 smithi115 podman[69497]: 2024-09-20 22:42:26.468703864 +0000 UTC m=+0.379569442 container init 3c6347d2a89122f3921d716a8bfb3a7c46a3c2387250313de367c2fc0d55a513 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_CLEAN=True) 2024-09-20T22:42:26.739 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:26 smithi115 podman[69497]: 2024-09-20 22:42:26.472047816 +0000 UTC m=+0.382913403 container start 3c6347d2a89122f3921d716a8bfb3a7c46a3c2387250313de367c2fc0d55a513 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:42:26.739 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:26 smithi115 podman[69497]: 2024-09-20 22:42:26.551940029 +0000 UTC m=+0.462805612 container attach 3c6347d2a89122f3921d716a8bfb3a7c46a3c2387250313de367c2fc0d55a513 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-20T22:42:26.739 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:26 smithi115 podman[69497]: 2024-09-20 22:42:26.654553459 +0000 UTC m=+0.565419042 container died 3c6347d2a89122f3921d716a8bfb3a7c46a3c2387250313de367c2fc0d55a513 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:42:27.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:26 smithi153 ceph-mon[56592]: pgmap v37: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:27.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:26 smithi153 ceph-mon[56592]: Health check failed: Reduced data availability: 13 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-09-20T22:42:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:26 smithi115 ceph-mon[64113]: pgmap v37: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:26 smithi115 ceph-mon[64113]: Health check failed: Reduced data availability: 13 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-09-20T22:42:27.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:26 smithi115 ceph-mon[66621]: pgmap v37: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:27.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:26 smithi115 ceph-mon[66621]: Health check failed: Reduced data availability: 13 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-09-20T22:42:27.648 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:27 smithi115 podman[69520]: 2024-09-20 22:42:27.214085443 +0000 UTC m=+0.553494432 container remove 3c6347d2a89122f3921d716a8bfb3a7c46a3c2387250313de367c2fc0d55a513 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-20T22:42:27.649 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:27 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.0.service: Deactivated successfully. 2024-09-20T22:42:27.649 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:27 smithi115 systemd[1]: Stopped Ceph osd.0 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:42:27.649 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:27 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.0.service: Consumed 6.583s CPU time. 2024-09-20T22:42:27.649 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:27 smithi115 systemd[1]: Starting Ceph osd.0 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:42:27.904 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:27 smithi115 podman[69620]: 2024-09-20 22:42:27.549505407 +0000 UTC m=+0.015840470 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:42:27.904 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:27 smithi115 podman[69620]: 2024-09-20 22:42:27.673313418 +0000 UTC m=+0.139648479 container create 257eafa5a234623615775b35d6458af818443b8929e7fa714ee9fc5e2293c382 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T22:42:28.404 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:27 smithi115 podman[69620]: 2024-09-20 22:42:27.984946035 +0000 UTC m=+0.451281098 container init 257eafa5a234623615775b35d6458af818443b8929e7fa714ee9fc5e2293c382 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-20T22:42:28.404 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:27 smithi115 podman[69620]: 2024-09-20 22:42:27.989214229 +0000 UTC m=+0.455549294 container start 257eafa5a234623615775b35d6458af818443b8929e7fa714ee9fc5e2293c382 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70) 2024-09-20T22:42:28.404 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:28 smithi115 podman[69620]: 2024-09-20 22:42:28.067609839 +0000 UTC m=+0.533944903 container attach 257eafa5a234623615775b35d6458af818443b8929e7fa714ee9fc5e2293c382 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-09-20T22:42:28.404 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:28 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate[69636]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:42:28.405 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:28 smithi115 bash[69620]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:42:28.405 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:28 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate[69636]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:42:28.405 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:28 smithi115 bash[69620]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:42:29.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:28 smithi153 ceph-mon[56592]: pgmap v39: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:29.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:28.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:42:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:28 smithi115 ceph-mon[64113]: pgmap v39: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:29.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:28 smithi115 ceph-mon[66621]: pgmap v39: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:29.791 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate[69636]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T22:42:29.791 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 bash[69620]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T22:42:29.791 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate[69636]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T22:42:29.791 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 bash[69620]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T22:42:29.792 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate[69636]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T22:42:29.792 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 bash[69620]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T22:42:30.122 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate[69636]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T22:42:30.122 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 bash[69620]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T22:42:30.122 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate[69636]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-20T22:42:30.122 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 bash[69620]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-20T22:42:30.122 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate[69636]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T22:42:30.123 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 bash[69620]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T22:42:30.123 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate[69636]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-20T22:42:30.123 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 bash[69620]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-20T22:42:30.123 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:29 smithi115 podman[69620]: 2024-09-20 22:42:29.820863209 +0000 UTC m=+2.287198265 container died 257eafa5a234623615775b35d6458af818443b8929e7fa714ee9fc5e2293c382 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-20T22:42:30.686 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:30 smithi115 podman[69784]: 2024-09-20 22:42:30.421495956 +0000 UTC m=+0.591341436 container remove 257eafa5a234623615775b35d6458af818443b8929e7fa714ee9fc5e2293c382 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-20T22:42:30.686 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:30 smithi115 podman[69842]: 2024-09-20 22:42:30.541404583 +0000 UTC m=+0.016023706 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:42:30.687 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:30 smithi115 podman[69842]: 2024-09-20 22:42:30.638825854 +0000 UTC m=+0.113445005 container create 98478f94c84ea77bb9dcf7f430a1c28097a4d1e60aa3ab7a7aa6d7f05af06279 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-20T22:42:30.967 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:30 smithi115 ceph-mon[64113]: pgmap v40: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:30.967 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:30 smithi115 ceph-mon[66621]: pgmap v40: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:30.968 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:30 smithi115 podman[69842]: 2024-09-20 22:42:30.874704758 +0000 UTC m=+0.349323875 container init 98478f94c84ea77bb9dcf7f430a1c28097a4d1e60aa3ab7a7aa6d7f05af06279 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:42:30.968 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:30 smithi115 podman[69842]: 2024-09-20 22:42:30.877987693 +0000 UTC m=+0.352606799 container start 98478f94c84ea77bb9dcf7f430a1c28097a4d1e60aa3ab7a7aa6d7f05af06279 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:42:30.968 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:30 smithi115 ceph-osd[69861]: -- 172.21.15.115:0/4077196362 <== mon.1 v2:172.21.15.115:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f1ad6afa40 con 0x55f1ae476000 2024-09-20T22:42:31.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:30 smithi153 ceph-mon[56592]: pgmap v40: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:31.404 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:30 smithi115 bash[69842]: 98478f94c84ea77bb9dcf7f430a1c28097a4d1e60aa3ab7a7aa6d7f05af06279 2024-09-20T22:42:31.404 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:30 smithi115 systemd[1]: Started Ceph osd.0 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:42:31.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:31 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:31.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:31.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:31.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:31 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:31.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:31.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:32.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:31 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:32.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:32.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:32.654 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:32 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0[69857]: 2024-09-20T22:42:32.269+0000 7f8e90206640 -1 Falling back to public interface 2024-09-20T22:42:32.820 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:42:33.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:32 smithi153 ceph-mon[56592]: pgmap v41: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 73/630 objects degraded (11.587%) 2024-09-20T22:42:33.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:32 smithi153 ceph-mon[56592]: Health check failed: Degraded data redundancy: 73/630 objects degraded (11.587%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T22:42:33.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:32 smithi153 ceph-mon[56592]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 56 pgs peering) 2024-09-20T22:42:33.152 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:32 smithi115 ceph-mon[64113]: pgmap v41: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 73/630 objects degraded (11.587%) 2024-09-20T22:42:33.152 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:32 smithi115 ceph-mon[64113]: Health check failed: Degraded data redundancy: 73/630 objects degraded (11.587%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T22:42:33.152 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:32 smithi115 ceph-mon[64113]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 56 pgs peering) 2024-09-20T22:42:33.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:32 smithi115 ceph-mon[66621]: pgmap v41: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 73/630 objects degraded (11.587%) 2024-09-20T22:42:33.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:32 smithi115 ceph-mon[66621]: Health check failed: Degraded data redundancy: 73/630 objects degraded (11.587%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T22:42:33.152 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:32 smithi115 ceph-mon[66621]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 56 pgs peering) 2024-09-20T22:42:33.510 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:42:33.510 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (2m) 54s ago 7m 19.1M - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:42:33.510 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (2m) 41s ago 7m 43.7M - dad864ee21e9 568386e3ebfd 2024-09-20T22:42:33.510 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (104s) 54s ago 6m 49.8M - 3.5 e1d6a67b021e e7840bcf555a 2024-09-20T22:42:33.510 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (94s) 41s ago 12m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:42:33.510 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (2m) 54s ago 13m 488M - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:42:33.510 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (80s) 54s ago 13m 37.5M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:42:33.511 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (46s) 41s ago 12m 20.6M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 f4dde7cfd242 2024-09-20T22:42:33.511 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (60s) 54s ago 12m 20.2M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 887b72a1971e 2024-09-20T22:42:33.511 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (2m) 54s ago 8m 15.3M - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:42:33.511 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (2m) 41s ago 8m 15.1M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:42:33.511 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 starting - - - 2736M 2024-09-20T22:42:33.511 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (11m) 54s ago 11m 53.5M 2736M 17.2.0 e1d6a67b021e b68eba671ad6 2024-09-20T22:42:33.511 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (10m) 54s ago 10m 48.1M 2736M 17.2.0 e1d6a67b021e a368d2632c3f 2024-09-20T22:42:33.511 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (10m) 54s ago 10m 53.8M 2736M 17.2.0 e1d6a67b021e d4ef32545846 2024-09-20T22:42:33.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (10m) 41s ago 10m 55.9M 2992M 17.2.0 e1d6a67b021e e9316b3d7317 2024-09-20T22:42:33.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (9m) 41s ago 9m 50.1M 2992M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:42:33.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (9m) 41s ago 9m 48.4M 2992M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:42:33.512 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (8m) 41s ago 8m 52.8M 2992M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:42:33.512 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (99s) 41s ago 8m 51.4M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:42:33.512 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (6m) 54s ago 6m 73.9M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:42:33.512 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (6m) 41s ago 6m 74.7M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:42:33.512 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (6m) 54s ago 6m 73.4M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:42:33.512 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (6m) 41s ago 6m 71.0M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:42:33.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:33 smithi153 ceph-mon[56592]: from='client.44208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:33.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:33.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:42:33.880 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:33 smithi115 ceph-mon[64113]: from='client.44208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:33.881 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:33 smithi115 ceph-mon[66621]: from='client.44208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:33.955 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:42:33.955 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:42:33.955 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:42:33.955 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:42:33.955 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:42:33.955 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:42:33.955 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:42:33.955 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:42:33.955 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7 2024-09-20T22:42:33.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:42:33.956 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:42:33.956 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:42:33.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:42:33.956 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:42:33.956 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-09-20T22:42:33.956 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 5 2024-09-20T22:42:33.956 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:42:33.956 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:42:34.296 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:42:34.297 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:42:34.297 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:42:34.297 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:42:34.297 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:42:34.297 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr", 2024-09-20T22:42:34.297 INFO:teuthology.orchestra.run.smithi115.stdout: "mon" 2024-09-20T22:42:34.297 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:42:34.297 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "5/23 daemons upgraded", 2024-09-20T22:42:34.297 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T22:42:34.297 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:42:34.297 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 73/630 objects degraded (11.587%), 25 pgs degraded 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi115.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi115.stdout: osd.0 (root=default,host=smithi115) is down 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi115.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 73/630 objects degraded (11.587%), 25 pgs degraded 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi115.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.0 is active+undersized+degraded, acting [7,1] 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.1 is active+undersized+degraded, acting [2,3] 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.4 is active+undersized+degraded, acting [1,7] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.8 is active+undersized+degraded, acting [7,1] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.f is active+undersized+degraded, acting [4,7] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.13 is active+undersized+degraded, acting [4,3] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.15 is active+undersized+degraded, acting [1,5] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.1d is active+undersized+degraded, acting [7,6] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-09-20T22:42:34.730 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-09-20T22:42:34.730 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-09-20T22:42:34.730 INFO:teuthology.orchestra.run.smithi115.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-09-20T22:42:34.730 INFO:teuthology.orchestra.run.smithi115.stdout: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-09-20T22:42:34.730 INFO:teuthology.orchestra.run.smithi115.stdout: pg 5.7 is active+undersized+degraded, acting [5,1] 2024-09-20T22:42:34.730 INFO:teuthology.orchestra.run.smithi115.stdout: pg 5.1d is active+undersized+degraded, acting [1,4] 2024-09-20T22:42:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[64113]: from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[64113]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[64113]: pgmap v42: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 73/630 objects degraded (11.587%) 2024-09-20T22:42:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3482096404' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:34.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/105758117' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:42:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[66621]: from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[66621]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[66621]: pgmap v42: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 73/630 objects degraded (11.587%) 2024-09-20T22:42:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/3482096404' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:34.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:34 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/105758117' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:42:35.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:34 smithi153 ceph-mon[56592]: from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:34 smithi153 ceph-mon[56592]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:34 smithi153 ceph-mon[56592]: pgmap v42: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 73/630 objects degraded (11.587%) 2024-09-20T22:42:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:34 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/3482096404' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:34 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:34 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:34 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/105758117' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:42:35.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:42:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:42:35] "GET /metrics HTTP/1.1" 200 38350 "" "Prometheus/2.43.0" 2024-09-20T22:42:36.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:35 smithi153 ceph-mon[56592]: from='client.44232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:36.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:35 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:36.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:35 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:35 smithi115 ceph-mon[64113]: from='client.44232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:35 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:35 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:36.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:35 smithi115 ceph-mon[66621]: from='client.44232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:42:36.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:35 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:36.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:35 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:36.155 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0[69857]: 2024-09-20T22:42:35.780+0000 7f8e90206640 -1 osd.0 0 read_superblock omap replica is missing. 2024-09-20T22:42:36.155 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0[69857]: 2024-09-20T22:42:35.808+0000 7f8e90206640 -1 osd.0 106 log_to_monitors true 2024-09-20T22:42:37.010 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:42:36 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0[69857]: 2024-09-20T22:42:36.797+0000 7f8e87fb1640 -1 osd.0 106 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:42:37.010 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:36 smithi115 ceph-mon[64113]: pgmap v43: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2024-09-20T22:42:37.010 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:36 smithi115 ceph-mon[64113]: from='osd.0 [v2:172.21.15.115:6802/3161504113,v1:172.21.15.115:6803/3161504113]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T22:42:37.010 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:36 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:37.011 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:36 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:37.011 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:36 smithi115 ceph-mon[66621]: pgmap v43: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2024-09-20T22:42:37.011 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:36 smithi115 ceph-mon[66621]: from='osd.0 [v2:172.21.15.115:6802/3161504113,v1:172.21.15.115:6803/3161504113]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T22:42:37.011 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:36 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:37.011 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:36 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:37.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:36 smithi153 ceph-mon[56592]: pgmap v43: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2024-09-20T22:42:37.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:36 smithi153 ceph-mon[56592]: from='osd.0 [v2:172.21.15.115:6802/3161504113,v1:172.21.15.115:6803/3161504113]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T22:42:37.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:36 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:37.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:36 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:38.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:37 smithi153 ceph-mon[56592]: from='osd.0 [v2:172.21.15.115:6802/3161504113,v1:172.21.15.115:6803/3161504113]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T22:42:38.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:37 smithi153 ceph-mon[56592]: osdmap e109: 8 total, 7 up, 8 in 2024-09-20T22:42:38.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:37 smithi153 ceph-mon[56592]: from='osd.0 [v2:172.21.15.115:6802/3161504113,v1:172.21.15.115:6803/3161504113]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:42:38.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:37 smithi115 ceph-mon[64113]: from='osd.0 [v2:172.21.15.115:6802/3161504113,v1:172.21.15.115:6803/3161504113]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T22:42:38.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:37 smithi115 ceph-mon[64113]: osdmap e109: 8 total, 7 up, 8 in 2024-09-20T22:42:38.053 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:37 smithi115 ceph-mon[64113]: from='osd.0 [v2:172.21.15.115:6802/3161504113,v1:172.21.15.115:6803/3161504113]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:42:38.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:37 smithi115 ceph-mon[66621]: from='osd.0 [v2:172.21.15.115:6802/3161504113,v1:172.21.15.115:6803/3161504113]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T22:42:38.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:37 smithi115 ceph-mon[66621]: osdmap e109: 8 total, 7 up, 8 in 2024-09-20T22:42:38.054 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:37 smithi115 ceph-mon[66621]: from='osd.0 [v2:172.21.15.115:6802/3161504113,v1:172.21.15.115:6803/3161504113]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:42:39.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:38 smithi153 ceph-mon[56592]: pgmap v45: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2024-09-20T22:42:39.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:38 smithi153 ceph-mon[56592]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:42:39.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:38 smithi153 ceph-mon[56592]: osd.0 [v2:172.21.15.115:6802/3161504113,v1:172.21.15.115:6803/3161504113] boot 2024-09-20T22:42:39.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:38 smithi153 ceph-mon[56592]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T22:42:39.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:38 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:42:39.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:38.659Z caller=alerting.go:366 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-20T22:42:39.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:38.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:42:39.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:38 smithi115 ceph-mon[66621]: pgmap v45: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2024-09-20T22:42:39.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:38 smithi115 ceph-mon[66621]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:42:39.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:38 smithi115 ceph-mon[66621]: osd.0 [v2:172.21.15.115:6802/3161504113,v1:172.21.15.115:6803/3161504113] boot 2024-09-20T22:42:39.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:38 smithi115 ceph-mon[66621]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T22:42:39.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:38 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:42:39.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:38 smithi115 ceph-mon[64113]: pgmap v45: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2024-09-20T22:42:39.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:38 smithi115 ceph-mon[64113]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:42:39.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:38 smithi115 ceph-mon[64113]: osd.0 [v2:172.21.15.115:6802/3161504113,v1:172.21.15.115:6803/3161504113] boot 2024-09-20T22:42:39.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:38 smithi115 ceph-mon[64113]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T22:42:39.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:38 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:42:39.788 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:42:39.661+0000 7f2588e7a640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:42:40.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[64113]: osdmap e111: 8 total, 8 up, 8 in 2024-09-20T22:42:40.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:40.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:40.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:40.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:42:40.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:40.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:42:40.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:40.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:40.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:40.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T22:42:40.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[66621]: osdmap e111: 8 total, 8 up, 8 in 2024-09-20T22:42:40.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:40.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:40.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:40.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:42:40.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:40.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:42:40.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:40.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:40.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:40.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:39 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T22:42:40.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:39 smithi153 ceph-mon[56592]: osdmap e111: 8 total, 8 up, 8 in 2024-09-20T22:42:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:39 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:39 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:39 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:39 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:42:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:39 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:39 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:42:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:39 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:39 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:39 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:42:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:39 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T22:42:41.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:40 smithi115 ceph-mon[64113]: pgmap v48: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2024-09-20T22:42:41.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:40 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T22:42:41.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:40 smithi115 ceph-mon[64113]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:42:41.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:40 smithi115 ceph-mon[66621]: pgmap v48: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2024-09-20T22:42:41.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:40 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T22:42:41.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:40 smithi115 ceph-mon[66621]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:42:41.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:40 smithi153 ceph-mon[56592]: pgmap v48: 161 pgs: 33 active+undersized, 25 active+undersized+degraded, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2024-09-20T22:42:41.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:40 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T22:42:41.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:40 smithi153 ceph-mon[56592]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:42:42.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:41 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:42.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:41 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:42.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:41 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:43.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:42 smithi115 ceph-mon[64113]: pgmap v49: 161 pgs: 13 active+undersized, 8 active+undersized+degraded, 140 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-09-20T22:42:43.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:42 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T22:42:43.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:42 smithi115 ceph-mon[66621]: pgmap v49: 161 pgs: 13 active+undersized, 8 active+undersized+degraded, 140 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-09-20T22:42:43.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:42 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T22:42:43.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:42 smithi153 ceph-mon[56592]: pgmap v49: 161 pgs: 13 active+undersized, 8 active+undersized+degraded, 140 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-09-20T22:42:43.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:42 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T22:42:43.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:43.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:42:44.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:43 smithi115 ceph-mon[64113]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded) 2024-09-20T22:42:44.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:43 smithi115 ceph-mon[64113]: Cluster is now healthy 2024-09-20T22:42:44.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:43 smithi115 ceph-mon[66621]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded) 2024-09-20T22:42:44.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:43 smithi115 ceph-mon[66621]: Cluster is now healthy 2024-09-20T22:42:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:43 smithi153 ceph-mon[56592]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded) 2024-09-20T22:42:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:43 smithi153 ceph-mon[56592]: Cluster is now healthy 2024-09-20T22:42:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:44 smithi115 ceph-mon[64113]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-09-20T22:42:45.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:44 smithi115 ceph-mon[66621]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-09-20T22:42:45.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:44 smithi153 ceph-mon[56592]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-09-20T22:42:45.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:42:45 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:42:45] "GET /metrics HTTP/1.1" 200 38350 "" "Prometheus/2.43.0" 2024-09-20T22:42:47.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:46 smithi115 ceph-mon[64113]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:47.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:46 smithi115 ceph-mon[66621]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:47.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:46 smithi153 ceph-mon[56592]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:49.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:48 smithi153 ceph-mon[56592]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T22:42:49.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:48 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:48.659Z caller=alerting.go:366 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-20T22:42:49.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:48 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:48.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:42:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:48 smithi115 ceph-mon[64113]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T22:42:49.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:48 smithi115 ceph-mon[66621]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T22:42:51.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:50 smithi153 ceph-mon[56592]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-09-20T22:42:51.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:50 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:51.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:50 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:51.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:50 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:50 smithi115 ceph-mon[64113]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-09-20T22:42:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:50 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:50 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:50 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:50 smithi115 ceph-mon[66621]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-09-20T22:42:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:50 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:50 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:50 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:52.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:51 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:51 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:52.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:51 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:53.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:52 smithi153 ceph-mon[56592]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:52 smithi115 ceph-mon[64113]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:53.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:52 smithi115 ceph-mon[66621]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:53.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:53.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:42:55.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:54 smithi153 ceph-mon[56592]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:55.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:54 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T22:42:55.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:54 smithi115 ceph-mon[64113]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:55.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:54 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T22:42:55.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:54 smithi115 ceph-mon[66621]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:55.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:54 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T22:42:55.376 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:42:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:42:55] "GET /metrics HTTP/1.1" 200 38289 "" "Prometheus/2.43.0" 2024-09-20T22:42:55.975 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:55 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T22:42:55.976 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:55 smithi115 ceph-mon[64113]: Upgrade: osd.1 is safe to restart 2024-09-20T22:42:55.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:55 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T22:42:55.976 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:55 smithi115 ceph-mon[66621]: Upgrade: osd.1 is safe to restart 2024-09-20T22:42:56.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:55 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T22:42:56.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:55 smithi153 ceph-mon[56592]: Upgrade: osd.1 is safe to restart 2024-09-20T22:42:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:56 smithi153 ceph-mon[56592]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:57.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:56 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:57.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:56 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T22:42:57.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:56 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:57.089 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:56 smithi115 ceph-mon[64113]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:57.089 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:56 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:57.089 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:56 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T22:42:57.089 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:56 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:57.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:56 smithi115 ceph-mon[66621]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:57.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:56 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:42:57.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:56 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T22:42:57.090 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:56 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:57.785 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:57 smithi115 ceph-mon[64113]: Upgrade: Updating osd.1 2024-09-20T22:42:57.786 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:57 smithi115 ceph-mon[64113]: Deploying daemon osd.1 on smithi115 2024-09-20T22:42:57.786 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:57 smithi115 ceph-mon[66621]: Upgrade: Updating osd.1 2024-09-20T22:42:57.786 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:57 smithi115 ceph-mon[66621]: Deploying daemon osd.1 on smithi115 2024-09-20T22:42:58.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:57 smithi153 ceph-mon[56592]: Upgrade: Updating osd.1 2024-09-20T22:42:58.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:57 smithi153 ceph-mon[56592]: Deploying daemon osd.1 on smithi115 2024-09-20T22:42:58.881 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:58 smithi115 ceph-mon[64113]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:58.882 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:58 smithi115 ceph-mon[66621]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:59.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:58 smithi153 ceph-mon[56592]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:42:59.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:42:58 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:42:58.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:42:59.154 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:42:58 smithi115 systemd[1]: Stopping Ceph osd.1 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:42:59.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:42:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1[33983]: 2024-09-20T22:42:59.156+0000 7f3e44951700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:42:59.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:42:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1[33983]: 2024-09-20T22:42:59.156+0000 7f3e44951700 -1 osd.1 111 *** Got signal Terminated *** 2024-09-20T22:42:59.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:42:59 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1[33983]: 2024-09-20T22:42:59.156+0000 7f3e44951700 -1 osd.1 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:43:00.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:42:59 smithi153 ceph-mon[56592]: osd.1 marked itself down and dead 2024-09-20T22:43:00.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:42:59 smithi115 ceph-mon[64113]: osd.1 marked itself down and dead 2024-09-20T22:43:00.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:42:59 smithi115 ceph-mon[66621]: osd.1 marked itself down and dead 2024-09-20T22:43:00.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:00 smithi115 podman[71875]: 2024-09-20 22:43:00.102518179 +0000 UTC m=+1.069587143 container died b68eba671ad6b9c02d4d5359af2ee9fba7f024ac1e1b1b0985a1e82f04487e28 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.openshift.expose-services=, vcs-type=git, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, architecture=x86_64, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_CLEAN=True, vendor=Red Hat, Inc.) 2024-09-20T22:43:00.719 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:00 smithi115 podman[71875]: 2024-09-20 22:43:00.492509982 +0000 UTC m=+1.459578941 container cleanup b68eba671ad6b9c02d4d5359af2ee9fba7f024ac1e1b1b0985a1e82f04487e28 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1, version=8, vcs-type=git, com.redhat.component=centos-stream-container, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, architecture=x86_64, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, ceph=True) 2024-09-20T22:43:00.719 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:00 smithi115 bash[71875]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1 2024-09-20T22:43:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:00 smithi153 ceph-mon[56592]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:00 smithi153 ceph-mon[56592]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:43:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:00 smithi153 ceph-mon[56592]: osdmap e112: 8 total, 7 up, 8 in 2024-09-20T22:43:01.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:00 smithi115 ceph-mon[64113]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:01.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:00 smithi115 ceph-mon[64113]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:43:01.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:00 smithi115 ceph-mon[64113]: osdmap e112: 8 total, 7 up, 8 in 2024-09-20T22:43:01.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:00 smithi115 ceph-mon[66621]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:01.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:00 smithi115 ceph-mon[66621]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:43:01.103 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:00 smithi115 ceph-mon[66621]: osdmap e112: 8 total, 7 up, 8 in 2024-09-20T22:43:01.103 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:00 smithi115 podman[71901]: 2024-09-20 22:43:00.777412202 +0000 UTC m=+0.673860500 container remove b68eba671ad6b9c02d4d5359af2ee9fba7f024ac1e1b1b0985a1e82f04487e28 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1, version=8, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, maintainer=Guillaume Abrioux , vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, ceph=True, com.redhat.component=centos-stream-container, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8) 2024-09-20T22:43:01.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:01 smithi115 podman[71992]: 2024-09-20 22:43:01.00401853 +0000 UTC m=+0.015954140 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:43:01.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:01 smithi115 podman[71992]: 2024-09-20 22:43:01.103082599 +0000 UTC m=+0.115018208 container create a7575545cf3a790db6df6327e260f2fecd1f0803c76478c83892bf16c2a87a37 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD) 2024-09-20T22:43:01.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:01 smithi115 podman[71992]: 2024-09-20 22:43:01.339123371 +0000 UTC m=+0.351058993 container init a7575545cf3a790db6df6327e260f2fecd1f0803c76478c83892bf16c2a87a37 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-20T22:43:01.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:01 smithi115 podman[71992]: 2024-09-20 22:43:01.342542868 +0000 UTC m=+0.354478480 container start a7575545cf3a790db6df6327e260f2fecd1f0803c76478c83892bf16c2a87a37 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70) 2024-09-20T22:43:01.731 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:01 smithi115 podman[71992]: 2024-09-20 22:43:01.413968271 +0000 UTC m=+0.425903892 container attach a7575545cf3a790db6df6327e260f2fecd1f0803c76478c83892bf16c2a87a37 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-09-20T22:43:01.731 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:01 smithi115 podman[71992]: 2024-09-20 22:43:01.53473358 +0000 UTC m=+0.546669187 container died a7575545cf3a790db6df6327e260f2fecd1f0803c76478c83892bf16c2a87a37 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T22:43:02.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:01 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:01 smithi153 ceph-mon[56592]: osdmap e113: 8 total, 7 up, 8 in 2024-09-20T22:43:02.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:01 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:02.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:01 smithi115 ceph-mon[64113]: osdmap e113: 8 total, 7 up, 8 in 2024-09-20T22:43:02.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:01 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:02.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:01 smithi115 ceph-mon[66621]: osdmap e113: 8 total, 7 up, 8 in 2024-09-20T22:43:02.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:02 smithi115 podman[72015]: 2024-09-20 22:43:02.268243944 +0000 UTC m=+0.727376873 container remove a7575545cf3a790db6df6327e260f2fecd1f0803c76478c83892bf16c2a87a37 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T22:43:02.655 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:02 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.1.service: Deactivated successfully. 2024-09-20T22:43:02.655 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:02 smithi115 systemd[1]: Stopped Ceph osd.1 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:43:02.655 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:02 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.1.service: Consumed 7.183s CPU time. 2024-09-20T22:43:02.655 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:02 smithi115 systemd[1]: Starting Ceph osd.1 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:43:03.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:02 smithi153 ceph-mon[56592]: pgmap v61: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:03.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:02 smithi153 ceph-mon[56592]: Health check failed: Reduced data availability: 10 pgs inactive, 17 pgs peering (PG_AVAILABILITY) 2024-09-20T22:43:03.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:02 smithi115 ceph-mon[64113]: pgmap v61: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:03.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:02 smithi115 ceph-mon[64113]: Health check failed: Reduced data availability: 10 pgs inactive, 17 pgs peering (PG_AVAILABILITY) 2024-09-20T22:43:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:02 smithi115 ceph-mon[66621]: pgmap v61: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:03.044 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:02 smithi115 ceph-mon[66621]: Health check failed: Reduced data availability: 10 pgs inactive, 17 pgs peering (PG_AVAILABILITY) 2024-09-20T22:43:03.044 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:02 smithi115 podman[72113]: 2024-09-20 22:43:02.613960417 +0000 UTC m=+0.015843045 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:43:03.044 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:02 smithi115 podman[72113]: 2024-09-20 22:43:02.719424168 +0000 UTC m=+0.121306787 container create ddea1cc30b67f0bc8961e4b90e490ca9952f0aea6490c9f7df1b91e4b9c7c99e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-20T22:43:03.045 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:02 smithi115 podman[72113]: 2024-09-20 22:43:02.930173359 +0000 UTC m=+0.332055982 container init ddea1cc30b67f0bc8961e4b90e490ca9952f0aea6490c9f7df1b91e4b9c7c99e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD) 2024-09-20T22:43:03.045 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:02 smithi115 podman[72113]: 2024-09-20 22:43:02.933476844 +0000 UTC m=+0.335359464 container start ddea1cc30b67f0bc8961e4b90e490ca9952f0aea6490c9f7df1b91e4b9c7c99e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-20T22:43:03.045 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:03 smithi115 podman[72113]: 2024-09-20 22:43:03.042276675 +0000 UTC m=+0.444159301 container attach ddea1cc30b67f0bc8961e4b90e490ca9952f0aea6490c9f7df1b91e4b9c7c99e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:43:03.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:03 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate[72128]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:43:03.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:03 smithi115 bash[72113]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:43:03.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:03 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate[72128]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:43:03.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:03 smithi115 bash[72113]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:43:03.778 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:03.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:43:04.143 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:03 smithi115 ceph-mon[64113]: Health check failed: Degraded data redundancy: 8/630 objects degraded (1.270%), 2 pgs degraded (PG_DEGRADED) 2024-09-20T22:43:04.144 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:03 smithi115 ceph-mon[66621]: Health check failed: Degraded data redundancy: 8/630 objects degraded (1.270%), 2 pgs degraded (PG_DEGRADED) 2024-09-20T22:43:04.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:03 smithi153 ceph-mon[56592]: Health check failed: Degraded data redundancy: 8/630 objects degraded (1.270%), 2 pgs degraded (PG_DEGRADED) 2024-09-20T22:43:04.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate[72128]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T22:43:04.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 bash[72113]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T22:43:04.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate[72128]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T22:43:04.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 bash[72113]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T22:43:04.790 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate[72128]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T22:43:04.790 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 bash[72113]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T22:43:04.790 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate[72128]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T22:43:04.790 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 bash[72113]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T22:43:04.790 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate[72128]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-20T22:43:04.790 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 bash[72113]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-20T22:43:04.791 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate[72128]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T22:43:04.791 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 bash[72113]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T22:43:04.791 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate[72128]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-20T22:43:04.791 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 bash[72113]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-20T22:43:04.791 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:04 smithi115 podman[72113]: 2024-09-20 22:43:04.455620849 +0000 UTC m=+1.857503475 container died ddea1cc30b67f0bc8961e4b90e490ca9952f0aea6490c9f7df1b91e4b9c7c99e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-20T22:43:05.084 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:04 smithi115 ceph-mon[64113]: pgmap v62: 161 pgs: 5 active+undersized, 56 peering, 2 active+undersized+degraded, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 8/630 objects degraded (1.270%) 2024-09-20T22:43:05.084 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:05.085 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:05.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:04 smithi115 ceph-mon[66621]: pgmap v62: 161 pgs: 5 active+undersized, 56 peering, 2 active+undersized+degraded, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 8/630 objects degraded (1.270%) 2024-09-20T22:43:05.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:05.085 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:05.085 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:43:05.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:04 smithi153 ceph-mon[56592]: pgmap v62: 161 pgs: 5 active+undersized, 56 peering, 2 active+undersized+degraded, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 8/630 objects degraded (1.270%) 2024-09-20T22:43:05.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:05.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:05.337 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:43:05 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:43:05] "GET /metrics HTTP/1.1" 200 38431 "" "Prometheus/2.43.0" 2024-09-20T22:43:05.652 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:05 smithi115 podman[72275]: 2024-09-20 22:43:05.336369823 +0000 UTC m=+0.874349691 container remove ddea1cc30b67f0bc8961e4b90e490ca9952f0aea6490c9f7df1b91e4b9c7c99e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-activate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0) 2024-09-20T22:43:05.653 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:05 smithi115 podman[72384]: 2024-09-20 22:43:05.444694924 +0000 UTC m=+0.016178989 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:43:05.653 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:05 smithi115 podman[72384]: 2024-09-20 22:43:05.545462573 +0000 UTC m=+0.116946621 container create 4c2f3b5efcd070a74e35cc2720476e854e4c07d8c1ea368aa5858825d745a456 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-20T22:43:05.763 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:43:05.763 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3m) 30s ago 7m 28.0M - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:43:05.763 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (2m) 73s ago 7m 43.7M - dad864ee21e9 568386e3ebfd 2024-09-20T22:43:05.763 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (2m) 30s ago 6m 52.2M - 3.5 e1d6a67b021e e7840bcf555a 2024-09-20T22:43:05.763 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (2m) 73s ago 12m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:43:05.763 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (2m) 30s ago 14m 498M - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:43:05.763 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (112s) 30s ago 14m 42.1M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:43:05.763 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (78s) 73s ago 12m 20.6M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 f4dde7cfd242 2024-09-20T22:43:05.763 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (92s) 30s ago 12m 36.0M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 887b72a1971e 2024-09-20T22:43:05.763 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3m) 30s ago 8m 15.5M - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:43:05.763 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (2m) 73s ago 8m 15.1M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:43:05.763 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (35s) 30s ago 12m 11.5M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 98478f94c84e 2024-09-20T22:43:05.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (11m) 30s ago 11m 53.3M 2736M 17.2.0 e1d6a67b021e b68eba671ad6 2024-09-20T22:43:05.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (11m) 30s ago 11m 49.5M 2736M 17.2.0 e1d6a67b021e a368d2632c3f 2024-09-20T22:43:05.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (11m) 30s ago 11m 55.4M 2736M 17.2.0 e1d6a67b021e d4ef32545846 2024-09-20T22:43:05.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (10m) 73s ago 10m 55.9M 2992M 17.2.0 e1d6a67b021e e9316b3d7317 2024-09-20T22:43:05.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (10m) 73s ago 10m 50.1M 2992M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:43:05.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (9m) 73s ago 9m 48.4M 2992M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:43:05.764 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (9m) 73s ago 9m 52.8M 2992M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:43:05.764 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (2m) 73s ago 8m 51.4M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:43:05.764 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (7m) 30s ago 7m 74.4M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:43:05.764 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (7m) 73s ago 7m 74.7M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:43:05.764 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (7m) 30s ago 7m 73.7M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:43:05.764 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (7m) 73s ago 7m 71.0M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:43:06.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:05 smithi115 ceph-mon[64113]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 17 pgs peering) 2024-09-20T22:43:06.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:05 smithi115 ceph-mon[66621]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 17 pgs peering) 2024-09-20T22:43:06.155 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:05 smithi115 podman[72384]: 2024-09-20 22:43:05.781044499 +0000 UTC m=+0.352528555 container init 4c2f3b5efcd070a74e35cc2720476e854e4c07d8c1ea368aa5858825d745a456 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-20T22:43:06.155 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:05 smithi115 podman[72384]: 2024-09-20 22:43:05.784435404 +0000 UTC m=+0.355919453 container start 4c2f3b5efcd070a74e35cc2720476e854e4c07d8c1ea368aa5858825d745a456 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:43:06.155 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:05 smithi115 ceph-osd[72427]: -- 172.21.15.115:0/2704445295 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ab0439da40 con 0x55ab05164000 2024-09-20T22:43:06.155 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:05 smithi115 bash[72384]: 4c2f3b5efcd070a74e35cc2720476e854e4c07d8c1ea368aa5858825d745a456 2024-09-20T22:43:06.155 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:05 smithi115 systemd[1]: Started Ceph osd.1 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:43:06.205 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:43:06.205 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:43:06.206 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:43:06.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:43:06.206 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:43:06.206 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:43:06.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:43:06.206 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:43:06.206 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-09-20T22:43:06.206 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T22:43:06.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:43:06.206 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:43:06.206 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:43:06.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:43:06.207 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:43:06.207 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-09-20T22:43:06.207 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 6 2024-09-20T22:43:06.207 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:43:06.207 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:43:06.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:05 smithi153 ceph-mon[56592]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 17 pgs peering) 2024-09-20T22:43:06.572 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:43:06.572 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:43:06.572 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:43:06.572 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:43:06.572 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:43:06.573 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr", 2024-09-20T22:43:06.573 INFO:teuthology.orchestra.run.smithi115.stdout: "mon" 2024-09-20T22:43:06.573 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:43:06.573 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "6/23 daemons upgraded", 2024-09-20T22:43:06.573 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T22:43:06.573 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:43:06.573 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:43:06.986 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 90/630 objects degraded (14.286%), 25 pgs degraded 2024-09-20T22:43:06.986 INFO:teuthology.orchestra.run.smithi115.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout: osd.1 (root=default,host=smithi115) is down 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 90/630 objects degraded (14.286%), 25 pgs degraded 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.2 is active+undersized+degraded, acting [5,6] 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.4 is active+undersized+degraded, acting [0,7] 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.8 is active+undersized+degraded, acting [7,0] 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.15 is active+undersized+degraded, acting [5,0] 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-09-20T22:43:06.987 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2024-09-20T22:43:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2024-09-20T22:43:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-09-20T22:43:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-09-20T22:43:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-09-20T22:43:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-09-20T22:43:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-09-20T22:43:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: pg 4.9 is active+undersized+degraded, acting [4,3] 2024-09-20T22:43:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: pg 4.d is active+undersized+degraded, acting [4,2] 2024-09-20T22:43:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: pg 4.e is active+undersized+degraded, acting [4,6] 2024-09-20T22:43:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: pg 4.1f is active+undersized+degraded, acting [6,5] 2024-09-20T22:43:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: pg 5.7 is active+undersized+degraded, acting [5,0] 2024-09-20T22:43:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: pg 5.1d is active+undersized+degraded, acting [4,0] 2024-09-20T22:43:06.988 INFO:teuthology.orchestra.run.smithi115.stdout: pg 6.1a is active+undersized+degraded, acting [4,5] 2024-09-20T22:43:07.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[64113]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:07.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[64113]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:07.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[64113]: pgmap v63: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 90/630 objects degraded (14.286%) 2024-09-20T22:43:07.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[64113]: from='client.44259 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:07.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:07.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:07.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/1121892284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:07.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[66621]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:07.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[66621]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:07.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[66621]: pgmap v63: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 90/630 objects degraded (14.286%) 2024-09-20T22:43:07.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[66621]: from='client.44259 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:07.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:07.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:07.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:06 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/1121892284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:07.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:06 smithi153 ceph-mon[56592]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:07.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:06 smithi153 ceph-mon[56592]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:07.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:06 smithi153 ceph-mon[56592]: pgmap v63: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 90/630 objects degraded (14.286%) 2024-09-20T22:43:07.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:06 smithi153 ceph-mon[56592]: from='client.44259 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:07.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:06 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:07.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:06 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:07.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:06 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/1121892284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:07.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:07 smithi115 ceph-mon[64113]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:07.850 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:07 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3929952404' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:43:07.850 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:07 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1[72420]: 2024-09-20T22:43:07.409+0000 7f0530788640 -1 Falling back to public interface 2024-09-20T22:43:07.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:07 smithi115 ceph-mon[66621]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:07.851 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:07 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/3929952404' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:43:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:07 smithi153 ceph-mon[56592]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:08.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:07 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/3929952404' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:43:09.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:08 smithi153 ceph-mon[56592]: pgmap v64: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 90/630 objects degraded (14.286%) 2024-09-20T22:43:09.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:08 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:08.659Z caller=alerting.go:366 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-20T22:43:09.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:08 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:08.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:43:09.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:08 smithi115 ceph-mon[64113]: pgmap v64: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 90/630 objects degraded (14.286%) 2024-09-20T22:43:09.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:08 smithi115 ceph-mon[66621]: pgmap v64: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 90/630 objects degraded (14.286%) 2024-09-20T22:43:10.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:10 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1[72420]: 2024-09-20T22:43:10.422+0000 7f0530788640 -1 osd.1 0 read_superblock omap replica is missing. 2024-09-20T22:43:10.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:10 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1[72420]: 2024-09-20T22:43:10.445+0000 7f0530788640 -1 osd.1 111 log_to_monitors true 2024-09-20T22:43:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:10 smithi115 ceph-mon[64113]: pgmap v65: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 90/630 objects degraded (14.286%) 2024-09-20T22:43:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:10 smithi115 ceph-mon[64113]: from='osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:43:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:10 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:10 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:11.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:10 smithi115 ceph-mon[66621]: pgmap v65: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 90/630 objects degraded (14.286%) 2024-09-20T22:43:11.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:10 smithi115 ceph-mon[66621]: from='osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:43:11.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:10 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:11.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:10 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:11.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:10 smithi153 ceph-mon[56592]: pgmap v65: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 90/630 objects degraded (14.286%) 2024-09-20T22:43:11.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:10 smithi153 ceph-mon[56592]: from='osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:43:11.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:10 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:11.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:10 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:11.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1[72420]: 2024-09-20T22:43:11.334+0000 7f052452b640 -1 osd.1 111 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:43:12.057 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:12.057 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[64113]: from='osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T22:43:12.057 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[64113]: osdmap e114: 8 total, 7 up, 8 in 2024-09-20T22:43:12.057 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[64113]: from='osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:43:12.057 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:12.057 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:12.057 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[64113]: from='osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408]' entity='osd.1' 2024-09-20T22:43:12.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:12.057 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[66621]: from='osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T22:43:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[66621]: osdmap e114: 8 total, 7 up, 8 in 2024-09-20T22:43:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[66621]: from='osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:43:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:12.058 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:11 smithi115 ceph-mon[66621]: from='osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408]' entity='osd.1' 2024-09-20T22:43:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:11 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:11 smithi153 ceph-mon[56592]: from='osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T22:43:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:11 smithi153 ceph-mon[56592]: osdmap e114: 8 total, 7 up, 8 in 2024-09-20T22:43:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:11 smithi153 ceph-mon[56592]: from='osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:43:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:11 smithi153 ceph-mon[56592]: from='osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408]' entity='osd.1' 2024-09-20T22:43:13.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:12 smithi115 ceph-mon[64113]: pgmap v67: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s; 90/630 objects degraded (14.286%) 2024-09-20T22:43:13.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:12 smithi115 ceph-mon[64113]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:43:13.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:12 smithi115 ceph-mon[64113]: osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408] boot 2024-09-20T22:43:13.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:12 smithi115 ceph-mon[64113]: osdmap e115: 8 total, 8 up, 8 in 2024-09-20T22:43:13.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:12 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:43:13.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:12 smithi115 ceph-mon[66621]: pgmap v67: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s; 90/630 objects degraded (14.286%) 2024-09-20T22:43:13.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:12 smithi115 ceph-mon[66621]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:43:13.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:12 smithi115 ceph-mon[66621]: osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408] boot 2024-09-20T22:43:13.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:12 smithi115 ceph-mon[66621]: osdmap e115: 8 total, 8 up, 8 in 2024-09-20T22:43:13.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:12 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:43:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:12 smithi153 ceph-mon[56592]: pgmap v67: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s; 90/630 objects degraded (14.286%) 2024-09-20T22:43:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:12 smithi153 ceph-mon[56592]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:43:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:12 smithi153 ceph-mon[56592]: osd.1 [v2:172.21.15.115:6810/4243655408,v1:172.21.15.115:6811/4243655408] boot 2024-09-20T22:43:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:12 smithi153 ceph-mon[56592]: osdmap e115: 8 total, 8 up, 8 in 2024-09-20T22:43:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:12 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:43:13.778 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:13.504Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:43:14.477 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:14 smithi115 ceph-mon[64113]: osdmap e116: 8 total, 8 up, 8 in 2024-09-20T22:43:14.477 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:14 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 90/630 objects degraded (14.286%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T22:43:14.477 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:14 smithi115 ceph-mon[64113]: pgmap v70: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 90/630 objects degraded (14.286%) 2024-09-20T22:43:14.477 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:14 smithi115 ceph-mon[66621]: osdmap e116: 8 total, 8 up, 8 in 2024-09-20T22:43:14.477 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:14 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 90/630 objects degraded (14.286%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T22:43:14.477 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:14 smithi115 ceph-mon[66621]: pgmap v70: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 90/630 objects degraded (14.286%) 2024-09-20T22:43:14.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:14 smithi153 ceph-mon[56592]: osdmap e116: 8 total, 8 up, 8 in 2024-09-20T22:43:14.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:14 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 90/630 objects degraded (14.286%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T22:43:14.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:14 smithi153 ceph-mon[56592]: pgmap v70: 161 pgs: 38 active+undersized, 25 active+undersized+degraded, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 90/630 objects degraded (14.286%) 2024-09-20T22:43:14.904 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:43:14 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:43:14.636+0000 7f2588e7a640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-20T22:43:15.509 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:43:15] "GET /metrics HTTP/1.1" 200 38431 "" "Prometheus/2.43.0" 2024-09-20T22:43:15.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:15.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:15.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:15.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:43:15.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:15.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:43:15.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:15.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:15.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:15.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T22:43:15.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:15 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T22:43:15.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:15 smithi153 ceph-mon[56592]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-20T22:43:15.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:15.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:15.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:15.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:43:15.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:15.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:43:15.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:15.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:15.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:15.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T22:43:15.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T22:43:15.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[64113]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-20T22:43:15.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:15.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:15.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:15.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:43:15.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:15.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:43:15.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:15.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:15.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:15.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T22:43:15.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T22:43:15.906 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:15 smithi115 ceph-mon[66621]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-20T22:43:16.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:16 smithi153 ceph-mon[56592]: pgmap v71: 161 pgs: 17 active+undersized, 9 active+undersized+degraded, 135 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 38/630 objects degraded (6.032%) 2024-09-20T22:43:16.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:16 smithi115 ceph-mon[64113]: pgmap v71: 161 pgs: 17 active+undersized, 9 active+undersized+degraded, 135 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 38/630 objects degraded (6.032%) 2024-09-20T22:43:16.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:16 smithi115 ceph-mon[66621]: pgmap v71: 161 pgs: 17 active+undersized, 9 active+undersized+degraded, 135 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 38/630 objects degraded (6.032%) 2024-09-20T22:43:18.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:17 smithi153 ceph-mon[56592]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 9 pgs degraded) 2024-09-20T22:43:18.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:17 smithi153 ceph-mon[56592]: Cluster is now healthy 2024-09-20T22:43:18.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:17 smithi115 ceph-mon[64113]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 9 pgs degraded) 2024-09-20T22:43:18.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:17 smithi115 ceph-mon[64113]: Cluster is now healthy 2024-09-20T22:43:18.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:17 smithi115 ceph-mon[66621]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 9 pgs degraded) 2024-09-20T22:43:18.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:17 smithi115 ceph-mon[66621]: Cluster is now healthy 2024-09-20T22:43:18.759 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:18.658Z caller=alerting.go:366 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-20T22:43:18.759 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:18.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:43:19.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:18 smithi153 ceph-mon[56592]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 753 B/s rd, 0 op/s 2024-09-20T22:43:19.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:18 smithi115 ceph-mon[64113]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 753 B/s rd, 0 op/s 2024-09-20T22:43:19.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:18 smithi115 ceph-mon[66621]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 753 B/s rd, 0 op/s 2024-09-20T22:43:21.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:20 smithi153 ceph-mon[56592]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:43:21.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:21.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:21.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:20 smithi115 ceph-mon[64113]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:43:21.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:21.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:21.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:20 smithi115 ceph-mon[66621]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:43:21.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:21.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:22.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:21 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:21 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:22.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:21 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:23.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:22 smithi153 ceph-mon[56592]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:43:23.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:22 smithi115 ceph-mon[64113]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:43:23.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:22 smithi115 ceph-mon[66621]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:43:23.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:23.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:43:25.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:24 smithi153 ceph-mon[56592]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2024-09-20T22:43:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:24 smithi115 ceph-mon[64113]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2024-09-20T22:43:25.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:24 smithi115 ceph-mon[66621]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 992 B/s rd, 0 op/s 2024-09-20T22:43:25.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:43:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:43:25] "GET /metrics HTTP/1.1" 200 38441 "" "Prometheus/2.43.0" 2024-09-20T22:43:26.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:25 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:25 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:26.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:25 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:27.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:26 smithi153 ceph-mon[56592]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:26 smithi115 ceph-mon[64113]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:27.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:26 smithi115 ceph-mon[66621]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:29.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:28 smithi153 ceph-mon[56592]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:29.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:28.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:43:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:28 smithi115 ceph-mon[64113]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:29.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:28 smithi115 ceph-mon[66621]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:30.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:29 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T22:43:30.043 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:29 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T22:43:30.043 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:29 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T22:43:30.981 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:30 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T22:43:30.981 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:30 smithi115 ceph-mon[64113]: Upgrade: osd.2 is safe to restart 2024-09-20T22:43:30.981 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:30 smithi115 ceph-mon[64113]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:30.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:30 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T22:43:30.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:30 smithi115 ceph-mon[66621]: Upgrade: osd.2 is safe to restart 2024-09-20T22:43:30.981 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:30 smithi115 ceph-mon[66621]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:31.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:30 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T22:43:31.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:30 smithi153 ceph-mon[56592]: Upgrade: osd.2 is safe to restart 2024-09-20T22:43:31.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:30 smithi153 ceph-mon[56592]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:32.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:31 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:32.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:32.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T22:43:32.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:32.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:31 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:32.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:32.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T22:43:32.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:31 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T22:43:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:32.931 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:32 smithi115 ceph-mon[64113]: Upgrade: Updating osd.2 2024-09-20T22:43:32.931 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:32 smithi115 ceph-mon[64113]: Deploying daemon osd.2 on smithi115 2024-09-20T22:43:32.931 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:32 smithi115 ceph-mon[64113]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:32.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:32 smithi115 ceph-mon[66621]: Upgrade: Updating osd.2 2024-09-20T22:43:32.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:32 smithi115 ceph-mon[66621]: Deploying daemon osd.2 on smithi115 2024-09-20T22:43:32.931 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:32 smithi115 ceph-mon[66621]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:33.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:32 smithi153 ceph-mon[56592]: Upgrade: Updating osd.2 2024-09-20T22:43:33.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:32 smithi153 ceph-mon[56592]: Deploying daemon osd.2 on smithi115 2024-09-20T22:43:33.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:32 smithi153 ceph-mon[56592]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:33.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:33.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:43:34.090 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:33 smithi115 systemd[1]: Stopping Ceph osd.2 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:43:34.404 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2[36880]: 2024-09-20T22:43:34.113+0000 7f258f6bf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:43:34.404 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2[36880]: 2024-09-20T22:43:34.113+0000 7f258f6bf700 -1 osd.2 116 *** Got signal Terminated *** 2024-09-20T22:43:34.404 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2[36880]: 2024-09-20T22:43:34.113+0000 7f258f6bf700 -1 osd.2 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:43:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:34 smithi153 ceph-mon[56592]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:34 smithi153 ceph-mon[56592]: osd.2 marked itself down and dead 2024-09-20T22:43:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:34 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:35.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:34 smithi115 ceph-mon[64113]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:35.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:34 smithi115 ceph-mon[64113]: osd.2 marked itself down and dead 2024-09-20T22:43:35.039 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:34 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:35.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:34 smithi115 ceph-mon[66621]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:43:35.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:34 smithi115 ceph-mon[66621]: osd.2 marked itself down and dead 2024-09-20T22:43:35.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:34 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:35.295 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:43:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:43:35] "GET /metrics HTTP/1.1" 200 38453 "" "Prometheus/2.43.0" 2024-09-20T22:43:35.599 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:35 smithi115 podman[74372]: 2024-09-20 22:43:35.293896281 +0000 UTC m=+1.280338650 container died a368d2632c3f67693c170658e8c951d656484b6134dea3ce9d84c5dfa512e037 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, RELEASE=HEAD, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, distribution-scope=public, vcs-type=git, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, version=8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, architecture=x86_64, io.openshift.tags=base centos centos-stream) 2024-09-20T22:43:35.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:35 smithi115 ceph-mon[64113]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:43:35.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:35 smithi115 ceph-mon[64113]: osdmap e117: 8 total, 7 up, 8 in 2024-09-20T22:43:35.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:35 smithi115 ceph-mon[64113]: osdmap e118: 8 total, 7 up, 8 in 2024-09-20T22:43:35.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:35 smithi115 ceph-mon[66621]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:43:35.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:35 smithi115 ceph-mon[66621]: osdmap e117: 8 total, 7 up, 8 in 2024-09-20T22:43:35.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:35 smithi115 ceph-mon[66621]: osdmap e118: 8 total, 7 up, 8 in 2024-09-20T22:43:35.905 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:35 smithi115 podman[74372]: 2024-09-20 22:43:35.7222398 +0000 UTC m=+1.708682169 container cleanup a368d2632c3f67693c170658e8c951d656484b6134dea3ce9d84c5dfa512e037 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2, release=754, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, vendor=Red Hat, Inc., GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True) 2024-09-20T22:43:35.905 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:35 smithi115 bash[74372]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2 2024-09-20T22:43:36.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:35 smithi153 ceph-mon[56592]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:43:36.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:35 smithi153 ceph-mon[56592]: osdmap e117: 8 total, 7 up, 8 in 2024-09-20T22:43:36.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:35 smithi153 ceph-mon[56592]: osdmap e118: 8 total, 7 up, 8 in 2024-09-20T22:43:36.392 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:35 smithi115 podman[74398]: 2024-09-20 22:43:35.990648897 +0000 UTC m=+0.693726123 container remove a368d2632c3f67693c170658e8c951d656484b6134dea3ce9d84c5dfa512e037 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2, io.openshift.tags=base centos centos-stream, name=centos-stream, ceph=True, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, distribution-scope=public, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vcs-type=git, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-09-20T22:43:36.654 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:36 smithi115 podman[74485]: 2024-09-20 22:43:36.294050434 +0000 UTC m=+0.016555895 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:43:36.654 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:36 smithi115 podman[74485]: 2024-09-20 22:43:36.416351931 +0000 UTC m=+0.138857383 container create 9ce07b1f96d1becd84572ae558f2b995e7d6f3f742ae8d7864e44f2c82ba91d2 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-deactivate, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:43:36.654 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:36 smithi115 podman[74485]: 2024-09-20 22:43:36.635458386 +0000 UTC m=+0.357963839 container init 9ce07b1f96d1becd84572ae558f2b995e7d6f3f742ae8d7864e44f2c82ba91d2 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True) 2024-09-20T22:43:36.654 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:36 smithi115 podman[74485]: 2024-09-20 22:43:36.638712628 +0000 UTC m=+0.361218074 container start 9ce07b1f96d1becd84572ae558f2b995e7d6f3f742ae8d7864e44f2c82ba91d2 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-20T22:43:37.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:36 smithi153 ceph-mon[56592]: pgmap v82: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:37.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:36 smithi153 ceph-mon[56592]: Health check failed: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-09-20T22:43:37.116 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:36 smithi115 ceph-mon[64113]: pgmap v82: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:37.116 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:36 smithi115 ceph-mon[64113]: Health check failed: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-09-20T22:43:37.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:36 smithi115 ceph-mon[66621]: pgmap v82: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:37.117 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:36 smithi115 ceph-mon[66621]: Health check failed: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-09-20T22:43:37.117 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:36 smithi115 podman[74485]: 2024-09-20 22:43:36.727056964 +0000 UTC m=+0.449562422 container attach 9ce07b1f96d1becd84572ae558f2b995e7d6f3f742ae8d7864e44f2c82ba91d2 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:43:37.117 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:36 smithi115 podman[74485]: 2024-09-20 22:43:36.827947978 +0000 UTC m=+0.550453426 container died 9ce07b1f96d1becd84572ae558f2b995e7d6f3f742ae8d7864e44f2c82ba91d2 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-20T22:43:37.349 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:43:37.821 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:37 smithi115 podman[74509]: 2024-09-20 22:43:37.497830937 +0000 UTC m=+0.664129084 container remove 9ce07b1f96d1becd84572ae558f2b995e7d6f3f742ae8d7864e44f2c82ba91d2 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-20T22:43:37.821 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:37 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.2.service: Deactivated successfully. 2024-09-20T22:43:37.822 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:37 smithi115 systemd[1]: Stopped Ceph osd.2 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:43:37.822 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:37 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.2.service: Consumed 6.053s CPU time. 2024-09-20T22:43:37.822 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:37 smithi115 systemd[1]: Starting Ceph osd.2 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:43:38.043 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:43:38.043 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (3m) 27s ago 8m 28.0M - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:43:38.043 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (3m) 105s ago 8m 43.7M - dad864ee21e9 568386e3ebfd 2024-09-20T22:43:38.043 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (2m) 27s ago 7m 52.3M - 3.5 e1d6a67b021e e7840bcf555a 2024-09-20T22:43:38.043 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (2m) 105s ago 13m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:43:38.043 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (3m) 27s ago 14m 501M - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:43:38.043 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2m) 27s ago 14m 45.1M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:43:38.043 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (110s) 105s ago 13m 20.6M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 f4dde7cfd242 2024-09-20T22:43:38.043 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2m) 27s ago 13m 39.3M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 887b72a1971e 2024-09-20T22:43:38.043 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (3m) 27s ago 9m 15.3M - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:43:38.043 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (3m) 105s ago 9m 15.1M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:43:38.044 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (67s) 27s ago 12m 43.4M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 98478f94c84e 2024-09-20T22:43:38.044 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (32s) 27s ago 12m 12.2M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 4c2f3b5efcd0 2024-09-20T22:43:38.044 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (11m) 27s ago 11m 51.2M 2736M 17.2.0 e1d6a67b021e a368d2632c3f 2024-09-20T22:43:38.044 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (11m) 27s ago 11m 56.0M 2736M 17.2.0 e1d6a67b021e d4ef32545846 2024-09-20T22:43:38.044 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (11m) 105s ago 11m 55.9M 2992M 17.2.0 e1d6a67b021e e9316b3d7317 2024-09-20T22:43:38.044 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (10m) 105s ago 10m 50.1M 2992M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:43:38.044 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (10m) 105s ago 10m 48.4M 2992M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:43:38.044 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (10m) 105s ago 10m 52.8M 2992M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:43:38.044 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (2m) 105s ago 9m 51.4M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:43:38.044 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (7m) 27s ago 7m 74.5M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:43:38.044 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (7m) 105s ago 7m 74.7M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:43:38.044 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (7m) 27s ago 7m 73.9M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:43:38.044 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (7m) 105s ago 7m 71.0M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:43:38.087 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:37 smithi115 ceph-mon[64113]: Health check failed: Degraded data redundancy: 24/630 objects degraded (3.810%), 3 pgs degraded (PG_DEGRADED) 2024-09-20T22:43:38.087 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:37 smithi115 ceph-mon[66621]: Health check failed: Degraded data redundancy: 24/630 objects degraded (3.810%), 3 pgs degraded (PG_DEGRADED) 2024-09-20T22:43:38.088 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:37 smithi115 podman[74679]: 2024-09-20 22:43:37.848011482 +0000 UTC m=+0.018463373 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:43:38.088 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:37 smithi115 podman[74679]: 2024-09-20 22:43:37.965692408 +0000 UTC m=+0.136144282 container create 948265fd727c9348b4f7c47607a8dcf06c4c46984a5d459f9fe7ef291df1e572 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70) 2024-09-20T22:43:38.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:37 smithi153 ceph-mon[56592]: Health check failed: Degraded data redundancy: 24/630 objects degraded (3.810%), 3 pgs degraded (PG_DEGRADED) 2024-09-20T22:43:38.345 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:38 smithi115 podman[74679]: 2024-09-20 22:43:38.193784436 +0000 UTC m=+0.364236318 container init 948265fd727c9348b4f7c47607a8dcf06c4c46984a5d459f9fe7ef291df1e572 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-20T22:43:38.345 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:38 smithi115 podman[74679]: 2024-09-20 22:43:38.197645104 +0000 UTC m=+0.368096986 container start 948265fd727c9348b4f7c47607a8dcf06c4c46984a5d459f9fe7ef291df1e572 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-20T22:43:38.345 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:38 smithi115 podman[74679]: 2024-09-20 22:43:38.285021967 +0000 UTC m=+0.455473854 container attach 948265fd727c9348b4f7c47607a8dcf06c4c46984a5d459f9fe7ef291df1e572 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-20T22:43:38.345 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:38 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate[74696]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:43:38.345 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:38 smithi115 bash[74679]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:43:38.345 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:38 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate[74696]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:43:38.345 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:38 smithi115 bash[74679]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:43:38.456 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:43:38.456 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:43:38.456 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:43:38.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:43:38.456 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:43:38.456 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:43:38.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:43:38.456 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:43:38.456 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-09-20T22:43:38.457 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:43:38.457 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:43:38.457 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:43:38.457 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:43:38.457 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:43:38.457 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:43:38.457 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-09-20T22:43:38.457 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 7 2024-09-20T22:43:38.457 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:43:38.457 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:43:38.793 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:43:38.793 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:43:38.793 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:43:38.793 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:43:38.793 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:43:38.794 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr", 2024-09-20T22:43:38.794 INFO:teuthology.orchestra.run.smithi115.stdout: "mon" 2024-09-20T22:43:38.794 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:43:38.794 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "7/23 daemons upgraded", 2024-09-20T22:43:38.794 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T22:43:38.794 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:43:38.795 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:43:39.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:38 smithi153 ceph-mon[56592]: from='client.44289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:39.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:38 smithi153 ceph-mon[56592]: pgmap v84: 161 pgs: 11 active+undersized, 33 peering, 3 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 24/630 objects degraded (3.810%) 2024-09-20T22:43:39.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:38 smithi153 ceph-mon[56592]: from='client.54229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:39.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:38 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/2389317045' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:39.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:38.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:43:39.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:38 smithi115 ceph-mon[64113]: from='client.44289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:39.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:38 smithi115 ceph-mon[64113]: pgmap v84: 161 pgs: 11 active+undersized, 33 peering, 3 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 24/630 objects degraded (3.810%) 2024-09-20T22:43:39.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:38 smithi115 ceph-mon[64113]: from='client.54229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:39.097 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:38 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/2389317045' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:39.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:38 smithi115 ceph-mon[66621]: from='client.44289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:39.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:38 smithi115 ceph-mon[66621]: pgmap v84: 161 pgs: 11 active+undersized, 33 peering, 3 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 24/630 objects degraded (3.810%) 2024-09-20T22:43:39.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:38 smithi115 ceph-mon[66621]: from='client.54229 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:39.098 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:38 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/2389317045' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:39.204 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 5 pgs inactive, 8 pgs peering; Degraded data redundancy: 24/630 objects degraded (3.810%), 3 pgs degraded 2024-09-20T22:43:39.204 INFO:teuthology.orchestra.run.smithi115.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T22:43:39.204 INFO:teuthology.orchestra.run.smithi115.stdout: osd.2 (root=default,host=smithi115) is down 2024-09-20T22:43:39.205 INFO:teuthology.orchestra.run.smithi115.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 5 pgs inactive, 8 pgs peering 2024-09-20T22:43:39.205 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.1 is stuck peering for 8m, current state peering, last acting [3,0] 2024-09-20T22:43:39.205 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.c is stuck peering for 60s, current state peering, last acting [5,0] 2024-09-20T22:43:39.205 INFO:teuthology.orchestra.run.smithi115.stdout: pg 2.e is stuck peering for 8m, current state peering, last acting [3,7] 2024-09-20T22:43:39.205 INFO:teuthology.orchestra.run.smithi115.stdout: pg 5.6 is stuck peering for 8m, current state peering, last acting [5,7] 2024-09-20T22:43:39.205 INFO:teuthology.orchestra.run.smithi115.stdout: pg 5.a is stuck peering for 8m, current state peering, last acting [4,3] 2024-09-20T22:43:39.205 INFO:teuthology.orchestra.run.smithi115.stdout: pg 5.d is stuck peering for 8m, current state peering, last acting [7,5] 2024-09-20T22:43:39.205 INFO:teuthology.orchestra.run.smithi115.stdout: pg 6.f is stuck peering for 7m, current state peering, last acting [3,4] 2024-09-20T22:43:39.205 INFO:teuthology.orchestra.run.smithi115.stdout: pg 6.14 is stuck peering for 7m, current state peering, last acting [4,7] 2024-09-20T22:43:39.205 INFO:teuthology.orchestra.run.smithi115.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 24/630 objects degraded (3.810%), 3 pgs degraded 2024-09-20T22:43:39.205 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-09-20T22:43:39.205 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-09-20T22:43:39.205 INFO:teuthology.orchestra.run.smithi115.stdout: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-09-20T22:43:39.394 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate[74696]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T22:43:39.394 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 bash[74679]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T22:43:39.394 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate[74696]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T22:43:39.394 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 bash[74679]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T22:43:39.394 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate[74696]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T22:43:39.654 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 bash[74679]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T22:43:39.654 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate[74696]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T22:43:39.654 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 bash[74679]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T22:43:39.654 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate[74696]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-20T22:43:39.654 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 bash[74679]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-20T22:43:39.655 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate[74696]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T22:43:39.655 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 bash[74679]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T22:43:39.655 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate[74696]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-20T22:43:39.655 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 bash[74679]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-20T22:43:39.655 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 podman[74679]: 2024-09-20 22:43:39.420415348 +0000 UTC m=+1.590867232 container died 948265fd727c9348b4f7c47607a8dcf06c4c46984a5d459f9fe7ef291df1e572 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-20T22:43:39.998 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:39 smithi115 ceph-mon[64113]: from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:39.998 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:39 smithi115 ceph-mon[64113]: from='client.44313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:39.998 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:39 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3252326084' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:43:39.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:39 smithi115 ceph-mon[66621]: from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:39.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:39 smithi115 ceph-mon[66621]: from='client.44313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:39.999 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:39 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/3252326084' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:43:40.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:39 smithi153 ceph-mon[56592]: from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:39 smithi153 ceph-mon[56592]: from='client.44313 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:43:40.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:39 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/3252326084' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:43:40.322 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:39 smithi115 podman[74912]: 2024-09-20 22:43:39.997089028 +0000 UTC m=+0.570383023 container remove 948265fd727c9348b4f7c47607a8dcf06c4c46984a5d459f9fe7ef291df1e572 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731) 2024-09-20T22:43:40.322 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:40 smithi115 podman[74971]: 2024-09-20 22:43:40.103851475 +0000 UTC m=+0.015814977 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:43:40.322 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:40 smithi115 podman[74971]: 2024-09-20 22:43:40.206456805 +0000 UTC m=+0.118420356 container create b90ea318f71233d5c74364f4f4ed199701d50c7250dae89008c4eaeed5f55b64 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-20T22:43:40.752 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:40 smithi115 podman[74971]: 2024-09-20 22:43:40.433869206 +0000 UTC m=+0.345832706 container init b90ea318f71233d5c74364f4f4ed199701d50c7250dae89008c4eaeed5f55b64 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-20T22:43:40.752 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:40 smithi115 podman[74971]: 2024-09-20 22:43:40.437101785 +0000 UTC m=+0.349065280 container start b90ea318f71233d5c74364f4f4ed199701d50c7250dae89008c4eaeed5f55b64 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2, ceph=True, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-20T22:43:40.752 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:40 smithi115 ceph-osd[74990]: -- 172.21.15.115:0/3229926507 <== mon.1 v2:172.21.15.115:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c9bcc7ba40 con 0x55c9bcc59c00 2024-09-20T22:43:40.752 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:40 smithi115 bash[74971]: b90ea318f71233d5c74364f4f4ed199701d50c7250dae89008c4eaeed5f55b64 2024-09-20T22:43:40.752 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:40 smithi115 systemd[1]: Started Ceph osd.2 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:43:41.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:40 smithi115 ceph-mon[64113]: pgmap v85: 161 pgs: 11 active+undersized, 33 peering, 3 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 24/630 objects degraded (3.810%) 2024-09-20T22:43:41.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:40 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:41.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:40 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:41.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:40 smithi115 ceph-mon[66621]: pgmap v85: 161 pgs: 11 active+undersized, 33 peering, 3 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 24/630 objects degraded (3.810%) 2024-09-20T22:43:41.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:40 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:41.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:40 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:41.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:40 smithi153 ceph-mon[56592]: pgmap v85: 161 pgs: 11 active+undersized, 33 peering, 3 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 24/630 objects degraded (3.810%) 2024-09-20T22:43:41.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:40 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:41.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:40 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:42.111 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:41 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:42.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:41 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:42.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:41 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:42.404 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:42 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2[74986]: 2024-09-20T22:43:42.110+0000 7f7764bc8640 -1 Falling back to public interface 2024-09-20T22:43:43.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:42 smithi115 ceph-mon[66621]: pgmap v86: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 55/630 objects degraded (8.730%) 2024-09-20T22:43:43.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:42 smithi115 ceph-mon[66621]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 8 pgs peering) 2024-09-20T22:43:43.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:42 smithi115 ceph-mon[64113]: pgmap v86: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 55/630 objects degraded (8.730%) 2024-09-20T22:43:43.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:42 smithi115 ceph-mon[64113]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 8 pgs peering) 2024-09-20T22:43:43.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:42 smithi153 ceph-mon[56592]: pgmap v86: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 55/630 objects degraded (8.730%) 2024-09-20T22:43:43.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:42 smithi153 ceph-mon[56592]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 8 pgs peering) 2024-09-20T22:43:43.778 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:43.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:43:44.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:43 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 55/630 objects degraded (8.730%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T22:43:44.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:43 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 55/630 objects degraded (8.730%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T22:43:44.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:43 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 55/630 objects degraded (8.730%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T22:43:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:44 smithi115 ceph-mon[64113]: pgmap v87: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 55/630 objects degraded (8.730%) 2024-09-20T22:43:45.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:44 smithi115 ceph-mon[66621]: pgmap v87: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 55/630 objects degraded (8.730%) 2024-09-20T22:43:45.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:44 smithi153 ceph-mon[56592]: pgmap v87: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 55/630 objects degraded (8.730%) 2024-09-20T22:43:45.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:43:45 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:43:45] "GET /metrics HTTP/1.1" 200 38453 "" "Prometheus/2.43.0" 2024-09-20T22:43:45.654 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:45 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2[74986]: 2024-09-20T22:43:45.368+0000 7f7764bc8640 -1 osd.2 0 read_superblock omap replica is missing. 2024-09-20T22:43:45.654 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:45 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2[74986]: 2024-09-20T22:43:45.390+0000 7f7764bc8640 -1 osd.2 116 log_to_monitors true 2024-09-20T22:43:46.634 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:46.634 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:46.635 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[64113]: from='osd.2 [v2:172.21.15.115:6818/1548759375,v1:172.21.15.115:6819/1548759375]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:43:46.635 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[64113]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:43:46.635 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[64113]: pgmap v88: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-09-20T22:43:46.635 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:46.635 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:46.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:46.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:46.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[66621]: from='osd.2 [v2:172.21.15.115:6818/1548759375,v1:172.21.15.115:6819/1548759375]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:43:46.635 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[66621]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:43:46.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[66621]: pgmap v88: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-09-20T22:43:46.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:46.636 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:46.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:46 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:46.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:46 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:46.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:46 smithi153 ceph-mon[56592]: from='osd.2 [v2:172.21.15.115:6818/1548759375,v1:172.21.15.115:6819/1548759375]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:43:46.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:46 smithi153 ceph-mon[56592]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:43:46.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:46 smithi153 ceph-mon[56592]: pgmap v88: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-09-20T22:43:46.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:46 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:46.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:46 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:46.904 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:43:46 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2[74986]: 2024-09-20T22:43:46.819+0000 7f775896b640 -1 osd.2 116 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:43:47.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:47 smithi115 ceph-mon[64113]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T22:43:47.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:47 smithi115 ceph-mon[64113]: osdmap e119: 8 total, 7 up, 8 in 2024-09-20T22:43:47.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:47 smithi115 ceph-mon[64113]: from='osd.2 [v2:172.21.15.115:6818/1548759375,v1:172.21.15.115:6819/1548759375]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:43:47.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:47 smithi115 ceph-mon[64113]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:43:47.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:47 smithi115 ceph-mon[64113]: from='osd.2 ' entity='osd.2' 2024-09-20T22:43:47.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:47 smithi115 ceph-mon[66621]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T22:43:47.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:47 smithi115 ceph-mon[66621]: osdmap e119: 8 total, 7 up, 8 in 2024-09-20T22:43:47.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:47 smithi115 ceph-mon[66621]: from='osd.2 [v2:172.21.15.115:6818/1548759375,v1:172.21.15.115:6819/1548759375]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:43:47.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:47 smithi115 ceph-mon[66621]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:43:47.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:47 smithi115 ceph-mon[66621]: from='osd.2 ' entity='osd.2' 2024-09-20T22:43:47.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:47 smithi153 ceph-mon[56592]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T22:43:47.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:47 smithi153 ceph-mon[56592]: osdmap e119: 8 total, 7 up, 8 in 2024-09-20T22:43:47.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:47 smithi153 ceph-mon[56592]: from='osd.2 [v2:172.21.15.115:6818/1548759375,v1:172.21.15.115:6819/1548759375]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:43:47.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:47 smithi153 ceph-mon[56592]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:43:47.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:47 smithi153 ceph-mon[56592]: from='osd.2 ' entity='osd.2' 2024-09-20T22:43:48.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:48 smithi115 ceph-mon[64113]: pgmap v90: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-09-20T22:43:48.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:48 smithi115 ceph-mon[64113]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:43:48.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:48 smithi115 ceph-mon[64113]: osd.2 [v2:172.21.15.115:6818/1548759375,v1:172.21.15.115:6819/1548759375] boot 2024-09-20T22:43:48.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:48 smithi115 ceph-mon[64113]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T22:43:48.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:48 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:43:48.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:48 smithi115 ceph-mon[66621]: pgmap v90: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-09-20T22:43:48.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:48 smithi115 ceph-mon[66621]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:43:48.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:48 smithi115 ceph-mon[66621]: osd.2 [v2:172.21.15.115:6818/1548759375,v1:172.21.15.115:6819/1548759375] boot 2024-09-20T22:43:48.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:48 smithi115 ceph-mon[66621]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T22:43:48.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:48 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:43:48.661 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:48 smithi153 ceph-mon[56592]: pgmap v90: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-09-20T22:43:48.661 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:48 smithi153 ceph-mon[56592]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:43:48.661 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:48 smithi153 ceph-mon[56592]: osd.2 [v2:172.21.15.115:6818/1548759375,v1:172.21.15.115:6819/1548759375] boot 2024-09-20T22:43:48.661 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:48 smithi153 ceph-mon[56592]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T22:43:48.661 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:48 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:43:49.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:48 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:48.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:43:49.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:43:49.315+0000 7f2588e7a640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-09-20T22:43:50.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[66621]: osdmap e121: 8 total, 8 up, 8 in 2024-09-20T22:43:50.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:50.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:43:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:43:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T22:43:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:50.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[64113]: osdmap e121: 8 total, 8 up, 8 in 2024-09-20T22:43:50.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:50.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:50.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:50.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:43:50.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:50.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:43:50.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:50.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:50.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:50.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T22:43:50.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:50.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:50.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:49 smithi153 ceph-mon[56592]: osdmap e121: 8 total, 8 up, 8 in 2024-09-20T22:43:50.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:50.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:50.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:50.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:43:50.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:50.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:43:50.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:50.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:50.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:43:50.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T22:43:50.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:43:50.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:50 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T22:43:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:50 smithi115 ceph-mon[64113]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-09-20T22:43:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:50 smithi115 ceph-mon[64113]: pgmap v93: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-09-20T22:43:51.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:50 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T22:43:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:50 smithi115 ceph-mon[66621]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-09-20T22:43:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:50 smithi115 ceph-mon[66621]: pgmap v93: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-09-20T22:43:51.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:50 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T22:43:51.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:50 smithi153 ceph-mon[56592]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-09-20T22:43:51.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:50 smithi153 ceph-mon[56592]: pgmap v93: 161 pgs: 34 active+undersized, 13 active+undersized+degraded, 114 active+clean; 585 KiB data, 147 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-09-20T22:43:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:51 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:52.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:51 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:52.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:51 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:52 smithi115 ceph-mon[64113]: pgmap v94: 161 pgs: 4 active+undersized, 1 active+undersized+degraded, 156 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5/630 objects degraded (0.794%) 2024-09-20T22:43:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:52 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 5/630 objects degraded (0.794%), 1 pg degraded (PG_DEGRADED) 2024-09-20T22:43:53.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:52 smithi115 ceph-mon[66621]: pgmap v94: 161 pgs: 4 active+undersized, 1 active+undersized+degraded, 156 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5/630 objects degraded (0.794%) 2024-09-20T22:43:53.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:52 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 5/630 objects degraded (0.794%), 1 pg degraded (PG_DEGRADED) 2024-09-20T22:43:53.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:52 smithi153 ceph-mon[56592]: pgmap v94: 161 pgs: 4 active+undersized, 1 active+undersized+degraded, 156 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 5/630 objects degraded (0.794%) 2024-09-20T22:43:53.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:52 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 5/630 objects degraded (0.794%), 1 pg degraded (PG_DEGRADED) 2024-09-20T22:43:53.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:53.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:43:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:53 smithi115 ceph-mon[64113]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/630 objects degraded (0.794%), 1 pg degraded) 2024-09-20T22:43:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:53 smithi115 ceph-mon[64113]: Cluster is now healthy 2024-09-20T22:43:54.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:53 smithi115 ceph-mon[66621]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/630 objects degraded (0.794%), 1 pg degraded) 2024-09-20T22:43:54.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:53 smithi115 ceph-mon[66621]: Cluster is now healthy 2024-09-20T22:43:54.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:53 smithi153 ceph-mon[56592]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/630 objects degraded (0.794%), 1 pg degraded) 2024-09-20T22:43:54.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:53 smithi153 ceph-mon[56592]: Cluster is now healthy 2024-09-20T22:43:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:54 smithi115 ceph-mon[64113]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 702 B/s rd, 0 op/s 2024-09-20T22:43:55.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:54 smithi115 ceph-mon[66621]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 702 B/s rd, 0 op/s 2024-09-20T22:43:55.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:54 smithi153 ceph-mon[56592]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 702 B/s rd, 0 op/s 2024-09-20T22:43:55.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:43:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:43:55] "GET /metrics HTTP/1.1" 200 38466 "" "Prometheus/2.43.0" 2024-09-20T22:43:57.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:56 smithi115 ceph-mon[64113]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:57.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:56 smithi115 ceph-mon[66621]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:57.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:56 smithi153 ceph-mon[56592]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:43:58.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:43:57 smithi115 ceph-mon[64113]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T22:43:58.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:43:57 smithi115 ceph-mon[66621]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T22:43:58.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:43:57 smithi153 ceph-mon[56592]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T22:43:59.028 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:43:58 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:43:58.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:44:01.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:00 smithi153 ceph-mon[56592]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-09-20T22:44:01.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:00 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:01.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:00 smithi115 ceph-mon[64113]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-09-20T22:44:01.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:00 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:01.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:00 smithi115 ceph-mon[66621]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-09-20T22:44:01.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:00 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:01 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:02.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:01 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:02.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:01 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:03.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:02 smithi153 ceph-mon[56592]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:02 smithi115 ceph-mon[64113]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:03.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:02 smithi115 ceph-mon[66621]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:03.778 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:03.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:44:04.979 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:04 smithi115 ceph-mon[64113]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:04.980 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T22:44:04.980 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:04.980 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:04.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:04 smithi115 ceph-mon[66621]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:04.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T22:44:04.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:04.980 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:05.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:04 smithi153 ceph-mon[56592]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T22:44:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:05.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:44:05 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:44:05] "GET /metrics HTTP/1.1" 200 38461 "" "Prometheus/2.43.0" 2024-09-20T22:44:06.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:05 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T22:44:06.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:05 smithi153 ceph-mon[56592]: Upgrade: osd.3 is safe to restart 2024-09-20T22:44:06.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:05 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:06.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:05 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T22:44:06.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:05 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:06.089 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:05 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T22:44:06.091 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:05 smithi115 ceph-mon[66621]: Upgrade: osd.3 is safe to restart 2024-09-20T22:44:06.112 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:05 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:06.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:05 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T22:44:06.115 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:05 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:06.115 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:05 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T22:44:06.116 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:05 smithi115 ceph-mon[64113]: Upgrade: osd.3 is safe to restart 2024-09-20T22:44:06.116 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:05 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:06.116 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:05 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T22:44:06.116 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:05 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:07.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:06 smithi153 ceph-mon[56592]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:07.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:06 smithi153 ceph-mon[56592]: Upgrade: Updating osd.3 2024-09-20T22:44:07.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:06 smithi153 ceph-mon[56592]: Deploying daemon osd.3 on smithi115 2024-09-20T22:44:07.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:06 smithi115 ceph-mon[66621]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:07.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:06 smithi115 ceph-mon[66621]: Upgrade: Updating osd.3 2024-09-20T22:44:07.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:06 smithi115 ceph-mon[66621]: Deploying daemon osd.3 on smithi115 2024-09-20T22:44:07.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:06 smithi115 ceph-mon[64113]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:07.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:06 smithi115 ceph-mon[64113]: Upgrade: Updating osd.3 2024-09-20T22:44:07.056 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:06 smithi115 ceph-mon[64113]: Deploying daemon osd.3 on smithi115 2024-09-20T22:44:08.750 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:08 smithi115 systemd[1]: Stopping Ceph osd.3 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:44:09.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:08 smithi153 ceph-mon[56592]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:09.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:08 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:08.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:44:09.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:08 smithi115 ceph-mon[64113]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:09.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:08 smithi115 ceph-mon[66621]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:09.045 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:08 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3[39848]: 2024-09-20T22:44:08.873+0000 7f3c9b3ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:44:09.046 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:08 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3[39848]: 2024-09-20T22:44:08.873+0000 7f3c9b3ea700 -1 osd.3 121 *** Got signal Terminated *** 2024-09-20T22:44:09.046 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:08 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3[39848]: 2024-09-20T22:44:08.873+0000 7f3c9b3ea700 -1 osd.3 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:44:09.569 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:44:09.830 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:09 smithi115 ceph-mon[64113]: osd.3 marked itself down and dead 2024-09-20T22:44:09.830 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:09 smithi115 ceph-mon[66621]: osd.3 marked itself down and dead 2024-09-20T22:44:10.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:09 smithi153 ceph-mon[56592]: osd.3 marked itself down and dead 2024-09-20T22:44:10.284 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:44:10.284 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4m) 24s ago 8m 28.0M - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (3m) 2m ago 8m 43.7M - dad864ee21e9 568386e3ebfd 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (3m) 24s ago 7m 52.5M - 3.5 e1d6a67b021e e7840bcf555a 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (3m) 2m ago 13m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (3m) 24s ago 15m 503M - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (2m) 24s ago 15m 46.9M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (2m) 2m ago 13m 20.6M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 f4dde7cfd242 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (2m) 24s ago 14m 40.0M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 887b72a1971e 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4m) 24s ago 9m 15.8M - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (3m) 2m ago 9m 15.1M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (99s) 24s ago 13m 44.0M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 98478f94c84e 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (64s) 24s ago 12m 53.3M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 4c2f3b5efcd0 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (30s) 24s ago 12m 11.7M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 b90ea318f712 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (12m) 24s ago 12m 56.7M 2736M 17.2.0 e1d6a67b021e d4ef32545846 2024-09-20T22:44:10.285 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (11m) 2m ago 11m 55.9M 2992M 17.2.0 e1d6a67b021e e9316b3d7317 2024-09-20T22:44:10.286 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (11m) 2m ago 11m 50.1M 2992M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:44:10.286 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (10m) 2m ago 10m 48.4M 2992M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:44:10.286 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (10m) 2m ago 10m 52.8M 2992M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:44:10.286 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (3m) 2m ago 9m 51.4M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:44:10.286 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (8m) 24s ago 8m 74.7M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:44:10.286 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (8m) 2m ago 8m 74.7M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:44:10.286 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (8m) 24s ago 8m 74.0M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:44:10.286 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (8m) 2m ago 8m 71.0M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:44:10.535 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:10 smithi115 podman[76863]: 2024-09-20 22:44:10.237624796 +0000 UTC m=+1.492363501 container died d4ef32545846e99e696537f3a8f7f47365866f5050b49a0c2a981221d6964551 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, release=754, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, version=8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.openshift.expose-services=, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD) 2024-09-20T22:44:10.698 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:44:10.698 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:44:10.698 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:44:10.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:44:10.698 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:44:10.698 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:44:10.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:44:10.698 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:44:10.698 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-09-20T22:44:10.699 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:44:10.699 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:44:10.699 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:44:10.699 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:44:10.699 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:44:10.699 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:44:10.699 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-09-20T22:44:10.699 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 8 2024-09-20T22:44:10.699 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:44:10.699 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:44:10.904 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:10 smithi115 podman[76863]: 2024-09-20 22:44:10.632763028 +0000 UTC m=+1.887501733 container cleanup d4ef32545846e99e696537f3a8f7f47365866f5050b49a0c2a981221d6964551 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vcs-type=git, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, ceph=True, GIT_CLEAN=True, version=8, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream) 2024-09-20T22:44:10.904 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:10 smithi115 bash[76863]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3 2024-09-20T22:44:10.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:10 smithi115 ceph-mon[64113]: from='client.44331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:10.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:10 smithi115 ceph-mon[64113]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:10.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:10 smithi115 ceph-mon[64113]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:44:10.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:10 smithi115 ceph-mon[64113]: osdmap e122: 8 total, 7 up, 8 in 2024-09-20T22:44:10.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:10 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/1675595428' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:10.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:10 smithi115 ceph-mon[66621]: from='client.44331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:10.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:10 smithi115 ceph-mon[66621]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:10.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:10 smithi115 ceph-mon[66621]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:44:10.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:10 smithi115 ceph-mon[66621]: osdmap e122: 8 total, 7 up, 8 in 2024-09-20T22:44:10.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:10 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/1675595428' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:11.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:10 smithi153 ceph-mon[56592]: from='client.44331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:10 smithi153 ceph-mon[56592]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:10 smithi153 ceph-mon[56592]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:44:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:10 smithi153 ceph-mon[56592]: osdmap e122: 8 total, 7 up, 8 in 2024-09-20T22:44:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:10 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/1675595428' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:11.070 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:44:11.070 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:44:11.071 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:44:11.071 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:44:11.071 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:44:11.071 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr", 2024-09-20T22:44:11.071 INFO:teuthology.orchestra.run.smithi115.stdout: "mon" 2024-09-20T22:44:11.071 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:44:11.071 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "8/23 daemons upgraded", 2024-09-20T22:44:11.071 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T22:44:11.071 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:44:11.071 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:44:11.299 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:10 smithi115 podman[76960]: 2024-09-20 22:44:10.909183319 +0000 UTC m=+0.672465262 container remove d4ef32545846e99e696537f3a8f7f47365866f5050b49a0c2a981221d6964551 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3, release=754, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.component=centos-stream-container, name=centos-stream, ceph=True, maintainer=Guillaume Abrioux , vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-09-20T22:44:11.509 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_WARN 1 osds down 2024-09-20T22:44:11.509 INFO:teuthology.orchestra.run.smithi115.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T22:44:11.509 INFO:teuthology.orchestra.run.smithi115.stdout: osd.3 (root=default,host=smithi115) is down 2024-09-20T22:44:11.550 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:11 smithi115 podman[77098]: 2024-09-20 22:44:11.200373633 +0000 UTC m=+0.022876631 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:44:11.551 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:11 smithi115 podman[77098]: 2024-09-20 22:44:11.326785221 +0000 UTC m=+0.149288198 container create 32d82348f22b79fcb81db94dd0003097ea3e1fcc7e6a383dbe3d495d66594273 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2) 2024-09-20T22:44:11.551 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:11 smithi115 podman[77098]: 2024-09-20 22:44:11.546005116 +0000 UTC m=+0.368508102 container init 32d82348f22b79fcb81db94dd0003097ea3e1fcc7e6a383dbe3d495d66594273 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-20T22:44:11.551 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:11 smithi115 podman[77098]: 2024-09-20 22:44:11.549400085 +0000 UTC m=+0.371903066 container start 32d82348f22b79fcb81db94dd0003097ea3e1fcc7e6a383dbe3d495d66594273 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-20T22:44:11.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:11 smithi115 ceph-mon[66621]: from='client.44337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:11.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:11 smithi115 ceph-mon[66621]: from='client.44343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:11.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:11 smithi115 ceph-mon[66621]: osdmap e123: 8 total, 7 up, 8 in 2024-09-20T22:44:11.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:11 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:11.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:11 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/1907029207' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:44:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:11 smithi115 ceph-mon[64113]: from='client.44337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:11 smithi115 ceph-mon[64113]: from='client.44343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:11 smithi115 ceph-mon[64113]: osdmap e123: 8 total, 7 up, 8 in 2024-09-20T22:44:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:11 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:11 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/1907029207' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:44:11.905 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:11 smithi115 podman[77098]: 2024-09-20 22:44:11.645980857 +0000 UTC m=+0.468483849 container attach 32d82348f22b79fcb81db94dd0003097ea3e1fcc7e6a383dbe3d495d66594273 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-20T22:44:11.905 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:11 smithi115 podman[77098]: 2024-09-20 22:44:11.738476344 +0000 UTC m=+0.560979329 container died 32d82348f22b79fcb81db94dd0003097ea3e1fcc7e6a383dbe3d495d66594273 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-20T22:44:12.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:11 smithi153 ceph-mon[56592]: from='client.44337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:12.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:11 smithi153 ceph-mon[56592]: from='client.44343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:12.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:11 smithi153 ceph-mon[56592]: osdmap e123: 8 total, 7 up, 8 in 2024-09-20T22:44:12.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:11 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:12.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:11 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/1907029207' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:44:12.654 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:12 smithi115 podman[77144]: 2024-09-20 22:44:12.333230645 +0000 UTC m=+0.584858482 container remove 32d82348f22b79fcb81db94dd0003097ea3e1fcc7e6a383dbe3d495d66594273 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-20T22:44:12.654 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:12 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.3.service: Deactivated successfully. 2024-09-20T22:44:12.654 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:12 smithi115 systemd[1]: Stopped Ceph osd.3 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:44:12.654 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:12 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.3.service: Consumed 7.865s CPU time. 2024-09-20T22:44:12.655 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:12 smithi115 systemd[1]: Starting Ceph osd.3 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:44:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:12 smithi153 ceph-mon[56592]: from='client.44355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:12 smithi153 ceph-mon[56592]: pgmap v106: 161 pgs: 12 active+undersized, 33 peering, 6 stale+active+clean, 7 active+undersized+degraded, 103 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 19/630 objects degraded (3.016%) 2024-09-20T22:44:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:12 smithi153 ceph-mon[56592]: Health check failed: Reduced data availability: 6 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-20T22:44:13.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:12 smithi153 ceph-mon[56592]: Health check failed: Degraded data redundancy: 19/630 objects degraded (3.016%), 7 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:13.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:12 smithi115 ceph-mon[64113]: from='client.44355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:13.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:12 smithi115 ceph-mon[64113]: pgmap v106: 161 pgs: 12 active+undersized, 33 peering, 6 stale+active+clean, 7 active+undersized+degraded, 103 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 19/630 objects degraded (3.016%) 2024-09-20T22:44:13.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:12 smithi115 ceph-mon[64113]: Health check failed: Reduced data availability: 6 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-20T22:44:13.046 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:12 smithi115 ceph-mon[64113]: Health check failed: Degraded data redundancy: 19/630 objects degraded (3.016%), 7 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:13.047 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:12 smithi115 podman[77243]: 2024-09-20 22:44:12.676844297 +0000 UTC m=+0.015770383 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:44:13.047 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:12 smithi115 podman[77243]: 2024-09-20 22:44:12.809470757 +0000 UTC m=+0.148396831 container create 4b92c3c97546b3cece072ac16b605e193e3dc47322c4ab5b1539f62fa32941c3 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-20T22:44:13.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:12 smithi115 ceph-mon[66621]: from='client.44355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:13.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:12 smithi115 ceph-mon[66621]: pgmap v106: 161 pgs: 12 active+undersized, 33 peering, 6 stale+active+clean, 7 active+undersized+degraded, 103 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 19/630 objects degraded (3.016%) 2024-09-20T22:44:13.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:12 smithi115 ceph-mon[66621]: Health check failed: Reduced data availability: 6 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-20T22:44:13.047 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:12 smithi115 ceph-mon[66621]: Health check failed: Degraded data redundancy: 19/630 objects degraded (3.016%), 7 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:13.404 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 podman[77243]: 2024-09-20 22:44:13.045293532 +0000 UTC m=+0.384219618 container init 4b92c3c97546b3cece072ac16b605e193e3dc47322c4ab5b1539f62fa32941c3 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:44:13.404 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 podman[77243]: 2024-09-20 22:44:13.048696294 +0000 UTC m=+0.387622375 container start 4b92c3c97546b3cece072ac16b605e193e3dc47322c4ab5b1539f62fa32941c3 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-20T22:44:13.405 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 podman[77243]: 2024-09-20 22:44:13.120112726 +0000 UTC m=+0.459038815 container attach 4b92c3c97546b3cece072ac16b605e193e3dc47322c4ab5b1539f62fa32941c3 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:44:13.405 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate[77259]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:44:13.405 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 bash[77243]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:44:13.405 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate[77259]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:44:13.405 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 bash[77243]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:44:13.778 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:13.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:44:13.904 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate[77259]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T22:44:13.904 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 bash[77243]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T22:44:13.904 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate[77259]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T22:44:13.904 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 bash[77243]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T22:44:14.258 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate[77259]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T22:44:14.258 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 bash[77243]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T22:44:14.258 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate[77259]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T22:44:14.258 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 bash[77243]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T22:44:14.258 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate[77259]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-20T22:44:14.259 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 bash[77243]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-20T22:44:14.259 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate[77259]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T22:44:14.259 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 bash[77243]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T22:44:14.259 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate[77259]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-20T22:44:14.259 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 bash[77243]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-20T22:44:14.259 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:13 smithi115 podman[77243]: 2024-09-20 22:44:13.966686253 +0000 UTC m=+1.305612342 container died 4b92c3c97546b3cece072ac16b605e193e3dc47322c4ab5b1539f62fa32941c3 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:44:14.838 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:14 smithi115 ceph-mon[64113]: pgmap v107: 161 pgs: 20 active+undersized, 33 peering, 13 active+undersized+degraded, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 42/630 objects degraded (6.667%) 2024-09-20T22:44:14.839 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:14 smithi115 podman[77388]: 2024-09-20 22:44:14.540438796 +0000 UTC m=+0.567513976 container remove 4b92c3c97546b3cece072ac16b605e193e3dc47322c4ab5b1539f62fa32941c3 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-activate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-20T22:44:14.839 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:14 smithi115 podman[77446]: 2024-09-20 22:44:14.749375883 +0000 UTC m=+0.112331422 container create 9eec1f15261b4c6f1e11e67e555e630d49f083f7502577fe21a3ca2efcc9a888 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:44:14.839 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:14 smithi115 podman[77446]: 2024-09-20 22:44:14.65279381 +0000 UTC m=+0.015749370 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:44:14.839 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:14 smithi115 ceph-mon[66621]: pgmap v107: 161 pgs: 20 active+undersized, 33 peering, 13 active+undersized+degraded, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 42/630 objects degraded (6.667%) 2024-09-20T22:44:15.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:14 smithi153 ceph-mon[56592]: pgmap v107: 161 pgs: 20 active+undersized, 33 peering, 13 active+undersized+degraded, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 42/630 objects degraded (6.667%) 2024-09-20T22:44:15.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:44:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:44:15] "GET /metrics HTTP/1.1" 200 38461 "" "Prometheus/2.43.0" 2024-09-20T22:44:15.405 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:14 smithi115 podman[77446]: 2024-09-20 22:44:14.968564513 +0000 UTC m=+0.331520057 container init 9eec1f15261b4c6f1e11e67e555e630d49f083f7502577fe21a3ca2efcc9a888 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T22:44:15.405 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:14 smithi115 podman[77446]: 2024-09-20 22:44:14.971999622 +0000 UTC m=+0.334955168 container start 9eec1f15261b4c6f1e11e67e555e630d49f083f7502577fe21a3ca2efcc9a888 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-20T22:44:15.405 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:14 smithi115 ceph-osd[77464]: -- 172.21.15.115:0/4188657159 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fc0635bd40 con 0x55fc070e2400 2024-09-20T22:44:15.405 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:14 smithi115 ceph-osd[77464]: -- 172.21.15.115:0/4188657159 <== mon.0 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fc0631ba40 con 0x55fc070e2400 2024-09-20T22:44:15.405 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:15 smithi115 bash[77446]: 9eec1f15261b4c6f1e11e67e555e630d49f083f7502577fe21a3ca2efcc9a888 2024-09-20T22:44:15.405 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:15 smithi115 systemd[1]: Started Ceph osd.3 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:44:16.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:16 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:16.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:16 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:16.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:16 smithi115 ceph-mon[64113]: pgmap v108: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 79/630 objects degraded (12.540%) 2024-09-20T22:44:16.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:16 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:16.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:16 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:16.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:16 smithi115 ceph-mon[66621]: pgmap v108: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 79/630 objects degraded (12.540%) 2024-09-20T22:44:16.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:16 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:16.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:16 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:16.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:16 smithi153 ceph-mon[56592]: pgmap v108: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 79/630 objects degraded (12.540%) 2024-09-20T22:44:16.904 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:16 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3[77460]: 2024-09-20T22:44:16.620+0000 7f65aa14d640 -1 Falling back to public interface 2024-09-20T22:44:17.387 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:17 smithi115 ceph-mon[64113]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 13 pgs peering) 2024-09-20T22:44:17.387 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:17 smithi115 ceph-mon[66621]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 13 pgs peering) 2024-09-20T22:44:17.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:17 smithi153 ceph-mon[56592]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 13 pgs peering) 2024-09-20T22:44:18.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:18 smithi115 ceph-mon[64113]: pgmap v109: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 79/630 objects degraded (12.540%) 2024-09-20T22:44:18.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:18 smithi115 ceph-mon[66621]: pgmap v109: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 79/630 objects degraded (12.540%) 2024-09-20T22:44:18.430 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:18 smithi153 ceph-mon[56592]: pgmap v109: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 79/630 objects degraded (12.540%) 2024-09-20T22:44:18.778 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:18.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:44:19.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:19 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 79/630 objects degraded (12.540%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:19.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:19 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 79/630 objects degraded (12.540%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:19.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:19 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 79/630 objects degraded (12.540%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:20.154 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3[77460]: 2024-09-20T22:44:19.879+0000 7f65aa14d640 -1 osd.3 0 read_superblock omap replica is missing. 2024-09-20T22:44:20.154 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:19 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3[77460]: 2024-09-20T22:44:19.907+0000 7f65aa14d640 -1 osd.3 121 log_to_monitors true 2024-09-20T22:44:20.955 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3[77460]: 2024-09-20T22:44:20.894+0000 7f65a1ef8640 -1 osd.3 121 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:44:20.955 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[64113]: pgmap v110: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 79/630 objects degraded (12.540%) 2024-09-20T22:44:20.955 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:20.955 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:20.955 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:20.955 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:20.955 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[64113]: from='osd.3 [v2:172.21.15.115:6826/3987120561,v1:172.21.15.115:6827/3987120561]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:44:20.955 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:20.956 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:20.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[66621]: pgmap v110: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 79/630 objects degraded (12.540%) 2024-09-20T22:44:20.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:20.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:20.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:20.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:20.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[66621]: from='osd.3 [v2:172.21.15.115:6826/3987120561,v1:172.21.15.115:6827/3987120561]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:44:20.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:20.956 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:21.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:20 smithi153 ceph-mon[56592]: pgmap v110: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 79/630 objects degraded (12.540%) 2024-09-20T22:44:21.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:21.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:21.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:21.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:21.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:20 smithi153 ceph-mon[56592]: from='osd.3 [v2:172.21.15.115:6826/3987120561,v1:172.21.15.115:6827/3987120561]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:44:21.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:21.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:22.147 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:21 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:22.147 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:21 smithi115 ceph-mon[64113]: from='osd.3 [v2:172.21.15.115:6826/3987120561,v1:172.21.15.115:6827/3987120561]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T22:44:22.147 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:21 smithi115 ceph-mon[64113]: osdmap e124: 8 total, 7 up, 8 in 2024-09-20T22:44:22.148 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:21 smithi115 ceph-mon[64113]: from='osd.3 [v2:172.21.15.115:6826/3987120561,v1:172.21.15.115:6827/3987120561]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:44:22.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:21 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:22.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:21 smithi115 ceph-mon[66621]: from='osd.3 [v2:172.21.15.115:6826/3987120561,v1:172.21.15.115:6827/3987120561]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T22:44:22.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:21 smithi115 ceph-mon[66621]: osdmap e124: 8 total, 7 up, 8 in 2024-09-20T22:44:22.148 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:21 smithi115 ceph-mon[66621]: from='osd.3 [v2:172.21.15.115:6826/3987120561,v1:172.21.15.115:6827/3987120561]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:44:22.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:21 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:21 smithi153 ceph-mon[56592]: from='osd.3 [v2:172.21.15.115:6826/3987120561,v1:172.21.15.115:6827/3987120561]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T22:44:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:21 smithi153 ceph-mon[56592]: osdmap e124: 8 total, 7 up, 8 in 2024-09-20T22:44:22.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:21 smithi153 ceph-mon[56592]: from='osd.3 [v2:172.21.15.115:6826/3987120561,v1:172.21.15.115:6827/3987120561]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-09-20T22:44:23.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:22 smithi115 ceph-mon[66621]: pgmap v112: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s; 79/630 objects degraded (12.540%) 2024-09-20T22:44:23.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:22 smithi115 ceph-mon[66621]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:44:23.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:22 smithi115 ceph-mon[66621]: osd.3 [v2:172.21.15.115:6826/3987120561,v1:172.21.15.115:6827/3987120561] boot 2024-09-20T22:44:23.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:22 smithi115 ceph-mon[66621]: osdmap e125: 8 total, 8 up, 8 in 2024-09-20T22:44:23.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:22 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:44:23.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:22 smithi115 ceph-mon[64113]: pgmap v112: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s; 79/630 objects degraded (12.540%) 2024-09-20T22:44:23.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:22 smithi115 ceph-mon[64113]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:44:23.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:22 smithi115 ceph-mon[64113]: osd.3 [v2:172.21.15.115:6826/3987120561,v1:172.21.15.115:6827/3987120561] boot 2024-09-20T22:44:23.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:22 smithi115 ceph-mon[64113]: osdmap e125: 8 total, 8 up, 8 in 2024-09-20T22:44:23.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:22 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:44:23.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:22 smithi153 ceph-mon[56592]: pgmap v112: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s; 79/630 objects degraded (12.540%) 2024-09-20T22:44:23.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:22 smithi153 ceph-mon[56592]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:44:23.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:22 smithi153 ceph-mon[56592]: osd.3 [v2:172.21.15.115:6826/3987120561,v1:172.21.15.115:6827/3987120561] boot 2024-09-20T22:44:23.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:22 smithi153 ceph-mon[56592]: osdmap e125: 8 total, 8 up, 8 in 2024-09-20T22:44:23.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:22 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:44:23.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:23.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:44:24.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:44:23.933+0000 7f2588e7a640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:44:24.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[64113]: osdmap e126: 8 total, 8 up, 8 in 2024-09-20T22:44:24.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[64113]: pgmap v115: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/630 objects degraded (12.540%) 2024-09-20T22:44:24.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:24.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:24.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:24.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:44:24.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:24.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[66621]: osdmap e126: 8 total, 8 up, 8 in 2024-09-20T22:44:24.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[66621]: pgmap v115: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/630 objects degraded (12.540%) 2024-09-20T22:44:24.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:24.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:24.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:24.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:44:24.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:23 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:24.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:23 smithi153 ceph-mon[56592]: osdmap e126: 8 total, 8 up, 8 in 2024-09-20T22:44:24.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:23 smithi153 ceph-mon[56592]: pgmap v115: 161 pgs: 41 active+undersized, 25 active+undersized+degraded, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/630 objects degraded (12.540%) 2024-09-20T22:44:24.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:23 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:24.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:23 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:24.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:23 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:24.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:23 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:44:24.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:23 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:25.202 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:44:25.202 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:25.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:25.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:25.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T22:44:25.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T22:44:25.203 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[64113]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:44:25.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:44:25.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:25.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:25.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:25.203 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T22:44:25.204 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T22:44:25.204 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:24 smithi115 ceph-mon[66621]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:44:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:24 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:44:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:24 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:24 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:24 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:24 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T22:44:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:24 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T22:44:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:24 smithi153 ceph-mon[56592]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:44:25.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:44:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:44:25] "GET /metrics HTTP/1.1" 200 38463 "" "Prometheus/2.43.0" 2024-09-20T22:44:26.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:25 smithi153 ceph-mon[56592]: pgmap v116: 161 pgs: 29 active+undersized, 18 active+undersized+degraded, 114 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/630 objects degraded (9.365%) 2024-09-20T22:44:26.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:25 smithi115 ceph-mon[64113]: pgmap v116: 161 pgs: 29 active+undersized, 18 active+undersized+degraded, 114 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/630 objects degraded (9.365%) 2024-09-20T22:44:26.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:25 smithi115 ceph-mon[66621]: pgmap v116: 161 pgs: 29 active+undersized, 18 active+undersized+degraded, 114 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/630 objects degraded (9.365%) 2024-09-20T22:44:27.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:26 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 59/630 objects degraded (9.365%), 18 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:27.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:26 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 59/630 objects degraded (9.365%), 18 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:27.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:26 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 59/630 objects degraded (9.365%), 18 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:28.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:27 smithi153 ceph-mon[56592]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:28.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:27 smithi115 ceph-mon[64113]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:28.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:27 smithi115 ceph-mon[66621]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:28.921 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:28.659Z caller=alerting.go:366 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-20T22:44:28.921 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:28.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:44:29.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:28 smithi153 ceph-mon[56592]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/630 objects degraded (9.365%), 18 pgs degraded) 2024-09-20T22:44:29.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:28 smithi153 ceph-mon[56592]: Cluster is now healthy 2024-09-20T22:44:29.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:28 smithi115 ceph-mon[64113]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/630 objects degraded (9.365%), 18 pgs degraded) 2024-09-20T22:44:29.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:28 smithi115 ceph-mon[64113]: Cluster is now healthy 2024-09-20T22:44:29.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:28 smithi115 ceph-mon[66621]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/630 objects degraded (9.365%), 18 pgs degraded) 2024-09-20T22:44:29.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:28 smithi115 ceph-mon[66621]: Cluster is now healthy 2024-09-20T22:44:30.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:29 smithi153 ceph-mon[56592]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:30.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:29 smithi115 ceph-mon[64113]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:30.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:29 smithi115 ceph-mon[66621]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:31.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:30 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:31.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:30 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:30 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:32.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:31 smithi153 ceph-mon[56592]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:32.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:31 smithi115 ceph-mon[64113]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:32.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:31 smithi115 ceph-mon[66621]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:33.779 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:33.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:44:35.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:34 smithi153 ceph-mon[56592]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:35.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:34 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:35.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:34 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:35.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:34 smithi115 ceph-mon[64113]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:35.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:34 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:35.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:34 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:35.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:34 smithi115 ceph-mon[66621]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:35.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:34 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:35.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:34 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:35.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:44:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:44:35] "GET /metrics HTTP/1.1" 200 38470 "" "Prometheus/2.43.0" 2024-09-20T22:44:36.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:35 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:35 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:36.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:35 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:37.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:36 smithi153 ceph-mon[56592]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:37.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:36 smithi115 ceph-mon[64113]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:37.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:36 smithi115 ceph-mon[66621]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:44:39.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:38 smithi153 ceph-mon[56592]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T22:44:39.029 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:38.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:44:39.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:38 smithi115 ceph-mon[64113]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T22:44:39.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:38 smithi115 ceph-mon[66621]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T22:44:40.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:39 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T22:44:40.133 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:39 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T22:44:40.133 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:39 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T22:44:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:40 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T22:44:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:40 smithi153 ceph-mon[56592]: Upgrade: osd.4 is safe to restart 2024-09-20T22:44:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:40 smithi153 ceph-mon[56592]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:40 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:40 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T22:44:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:40 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:41.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:40 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T22:44:41.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:40 smithi115 ceph-mon[66621]: Upgrade: osd.4 is safe to restart 2024-09-20T22:44:41.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:40 smithi115 ceph-mon[66621]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:41.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:40 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:41.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:40 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T22:44:41.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:40 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:41.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:40 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T22:44:41.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:40 smithi115 ceph-mon[64113]: Upgrade: osd.4 is safe to restart 2024-09-20T22:44:41.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:40 smithi115 ceph-mon[64113]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:41.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:40 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:41.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:40 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T22:44:41.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:40 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:41.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:41 smithi153 ceph-mon[56592]: Upgrade: Updating osd.4 2024-09-20T22:44:41.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:41 smithi153 ceph-mon[56592]: Deploying daemon osd.4 on smithi153 2024-09-20T22:44:41.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:41 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:41.874 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:44:42.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:41 smithi115 ceph-mon[64113]: Upgrade: Updating osd.4 2024-09-20T22:44:42.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:41 smithi115 ceph-mon[64113]: Deploying daemon osd.4 on smithi153 2024-09-20T22:44:42.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:41 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:42.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:41 smithi115 ceph-mon[66621]: Upgrade: Updating osd.4 2024-09-20T22:44:42.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:41 smithi115 ceph-mon[66621]: Deploying daemon osd.4 on smithi153 2024-09-20T22:44:42.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:41 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:42.525 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:44:42.525 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (4m) 22s ago 9m 26.0M - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:44:42.525 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (4m) 2m ago 9m 43.7M - dad864ee21e9 568386e3ebfd 2024-09-20T22:44:42.525 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (3m) 22s ago 8m 52.7M - 3.5 e1d6a67b021e e7840bcf555a 2024-09-20T22:44:42.525 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (3m) 2m ago 14m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:44:42.525 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (4m) 22s ago 15m 507M - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:44:42.525 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (3m) 22s ago 15m 46.4M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:44:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (2m) 2m ago 14m 20.6M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 f4dde7cfd242 2024-09-20T22:44:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3m) 22s ago 14m 38.3M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 887b72a1971e 2024-09-20T22:44:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (4m) 22s ago 10m 14.7M - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:44:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (4m) 2m ago 10m 15.1M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:44:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2m) 22s ago 13m 45.3M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 98478f94c84e 2024-09-20T22:44:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (97s) 22s ago 13m 53.7M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 4c2f3b5efcd0 2024-09-20T22:44:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (62s) 22s ago 13m 50.0M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 b90ea318f712 2024-09-20T22:44:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (27s) 22s ago 12m 11.1M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 9eec1f15261b 2024-09-20T22:44:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (12m) 2m ago 12m 55.9M 2992M 17.2.0 e1d6a67b021e e9316b3d7317 2024-09-20T22:44:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (11m) 2m ago 11m 50.1M 2992M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:44:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (11m) 2m ago 11m 48.4M 2992M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:44:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (11m) 2m ago 11m 52.8M 2992M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:44:42.526 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (3m) 2m ago 10m 51.4M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:44:42.527 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (8m) 22s ago 8m 74.9M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:44:42.527 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (8m) 2m ago 8m 74.7M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:44:42.527 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (8m) 22s ago 8m 74.1M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:44:42.527 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (8m) 2m ago 8m 71.0M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:44:42.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:42 smithi115 ceph-mon[64113]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:42.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:42 smithi115 ceph-mon[64113]: from='client.44373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:42.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:42 smithi115 ceph-mon[66621]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:42.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:42 smithi115 ceph-mon[66621]: from='client.44373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:42.920 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:44:42.920 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:44:42.920 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:44:42.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:44:42.920 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:44:42.920 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:44:42.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:44:42.921 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:44:42.921 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-09-20T22:44:42.921 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 4 2024-09-20T22:44:42.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:44:42.921 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:44:42.921 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:44:42.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:44:42.921 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:44:42.921 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-09-20T22:44:42.921 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 9 2024-09-20T22:44:42.921 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:44:42.921 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:44:43.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:42 smithi153 ceph-mon[56592]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:44:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:42 smithi153 ceph-mon[56592]: from='client.44373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:43.252 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:44:43.252 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:44:43.253 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:44:43.253 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:44:43.253 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:44:43.253 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr", 2024-09-20T22:44:43.253 INFO:teuthology.orchestra.run.smithi115.stdout: "mon" 2024-09-20T22:44:43.253 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:44:43.253 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "9/23 daemons upgraded", 2024-09-20T22:44:43.253 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T22:44:43.253 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:44:43.253 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:44:43.649 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:44:43.697 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:43.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:44:43.697 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:43 smithi153 systemd[1]: Stopping Ceph osd.4 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:44:43.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:43 smithi115 ceph-mon[64113]: from='client.44379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:43.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:43 smithi115 ceph-mon[64113]: from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:43.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:43 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/725489830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:43.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:43 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3546078250' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:44:43.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:43 smithi115 ceph-mon[64113]: osd.4 marked itself down and dead 2024-09-20T22:44:43.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:43 smithi115 ceph-mon[66621]: from='client.44379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:43.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:43 smithi115 ceph-mon[66621]: from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:43.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:43 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/725489830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:43.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:43 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/3546078250' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:44:43.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:43 smithi115 ceph-mon[66621]: osd.4 marked itself down and dead 2024-09-20T22:44:44.029 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[27910]: 2024-09-20T22:44:43.694+0000 7fdf4f325700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:44:44.029 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[27910]: 2024-09-20T22:44:43.694+0000 7fdf4f325700 -1 osd.4 126 *** Got signal Terminated *** 2024-09-20T22:44:44.029 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:43 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[27910]: 2024-09-20T22:44:43.694+0000 7fdf4f325700 -1 osd.4 126 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:44:44.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:43 smithi153 ceph-mon[56592]: from='client.44379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:44.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:43 smithi153 ceph-mon[56592]: from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:44.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:43 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/725489830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:44.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:43 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/3546078250' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:44:44.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:43 smithi153 ceph-mon[56592]: osd.4 marked itself down and dead 2024-09-20T22:44:44.528 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:44 smithi153 podman[59354]: 2024-09-20 22:44:44.244348537 +0000 UTC m=+0.681978463 container died e9316b3d731723ad63ffcf0358a4cbfbd9c02ecbd90ee099e9c39882895ad364 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vendor=Red Hat, Inc., distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_BRANCH=HEAD, ceph=True, io.openshift.expose-services=, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, release=754, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-09-20T22:44:45.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:44 smithi153 ceph-mon[56592]: from='client.44397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:45.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:44 smithi153 ceph-mon[56592]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:45.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:44 smithi153 ceph-mon[56592]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:44:45.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:44 smithi153 ceph-mon[56592]: osdmap e127: 8 total, 7 up, 8 in 2024-09-20T22:44:45.029 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:44 smithi153 podman[59354]: 2024-09-20 22:44:44.753201558 +0000 UTC m=+1.190831473 container cleanup e9316b3d731723ad63ffcf0358a4cbfbd9c02ecbd90ee099e9c39882895ad364 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, name=centos-stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, vcs-type=git, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, release=754) 2024-09-20T22:44:45.029 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:44 smithi153 bash[59354]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4 2024-09-20T22:44:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:44 smithi115 ceph-mon[64113]: from='client.44397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:44 smithi115 ceph-mon[64113]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:44 smithi115 ceph-mon[64113]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:44:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:44 smithi115 ceph-mon[64113]: osdmap e127: 8 total, 7 up, 8 in 2024-09-20T22:44:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:44 smithi115 ceph-mon[66621]: from='client.44397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:44:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:44 smithi115 ceph-mon[66621]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:44:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:44 smithi115 ceph-mon[66621]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:44:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:44 smithi115 ceph-mon[66621]: osdmap e127: 8 total, 7 up, 8 in 2024-09-20T22:44:45.378 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:45 smithi153 podman[59379]: 2024-09-20 22:44:45.079826341 +0000 UTC m=+0.831940933 container remove e9316b3d731723ad63ffcf0358a4cbfbd9c02ecbd90ee099e9c39882895ad364 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4, GIT_CLEAN=True, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, distribution-scope=public, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, ceph=True, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, vcs-type=git, name=centos-stream, release=754, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870) 2024-09-20T22:44:45.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:44:45 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:44:45] "GET /metrics HTTP/1.1" 200 38470 "" "Prometheus/2.43.0" 2024-09-20T22:44:45.709 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:45 smithi153 podman[59471]: 2024-09-20 22:44:45.280048949 +0000 UTC m=+0.015839357 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:44:45.710 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:45 smithi153 podman[59471]: 2024-09-20 22:44:45.405660916 +0000 UTC m=+0.141451316 container create 6874a5efd39be17f67cac9caff887b3ff0b9f328815b7d5c1b14d13b8e2dbb6e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-20T22:44:45.710 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:45 smithi153 podman[59471]: 2024-09-20 22:44:45.600206498 +0000 UTC m=+0.335996903 container init 6874a5efd39be17f67cac9caff887b3ff0b9f328815b7d5c1b14d13b8e2dbb6e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=) 2024-09-20T22:44:45.710 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:45 smithi153 podman[59471]: 2024-09-20 22:44:45.603606416 +0000 UTC m=+0.339396822 container start 6874a5efd39be17f67cac9caff887b3ff0b9f328815b7d5c1b14d13b8e2dbb6e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_CLEAN=True, ceph=True, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-20T22:44:45.710 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:45 smithi153 podman[59471]: 2024-09-20 22:44:45.708515394 +0000 UTC m=+0.444305813 container attach 6874a5efd39be17f67cac9caff887b3ff0b9f328815b7d5c1b14d13b8e2dbb6e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-20T22:44:46.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:45 smithi153 ceph-mon[56592]: osdmap e128: 8 total, 7 up, 8 in 2024-09-20T22:44:46.029 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:45 smithi153 podman[59471]: 2024-09-20 22:44:45.794549256 +0000 UTC m=+0.530339665 container died 6874a5efd39be17f67cac9caff887b3ff0b9f328815b7d5c1b14d13b8e2dbb6e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:44:46.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:45 smithi115 ceph-mon[64113]: osdmap e128: 8 total, 7 up, 8 in 2024-09-20T22:44:46.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:45 smithi115 ceph-mon[66621]: osdmap e128: 8 total, 7 up, 8 in 2024-09-20T22:44:46.779 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:46 smithi153 podman[59494]: 2024-09-20 22:44:46.437140138 +0000 UTC m=+0.636432173 container remove 6874a5efd39be17f67cac9caff887b3ff0b9f328815b7d5c1b14d13b8e2dbb6e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-20T22:44:46.779 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:46 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.4.service: Deactivated successfully. 2024-09-20T22:44:46.779 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:46 smithi153 systemd[1]: Stopped Ceph osd.4 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:44:46.779 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:46 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.4.service: Consumed 8.431s CPU time. 2024-09-20T22:44:46.779 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:46 smithi153 systemd[1]: Starting Ceph osd.4 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:44:47.100 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:46 smithi153 ceph-mon[56592]: pgmap v128: 161 pgs: 22 active+undersized, 14 stale+active+clean, 11 active+undersized+degraded, 114 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-09-20T22:44:47.101 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:46 smithi153 ceph-mon[56592]: Health check failed: Degraded data redundancy: 55/630 objects degraded (8.730%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:47.101 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:46 smithi153 podman[59596]: 2024-09-20 22:44:46.771491907 +0000 UTC m=+0.015919094 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:44:47.101 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:46 smithi153 podman[59596]: 2024-09-20 22:44:46.896673539 +0000 UTC m=+0.141100715 container create 56c41ebb736d4893fa8b0d19ed6a70af576e6cff3c6e803dfd831e33fddef536 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-20T22:44:47.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:46 smithi115 ceph-mon[64113]: pgmap v128: 161 pgs: 22 active+undersized, 14 stale+active+clean, 11 active+undersized+degraded, 114 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-09-20T22:44:47.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:46 smithi115 ceph-mon[64113]: Health check failed: Degraded data redundancy: 55/630 objects degraded (8.730%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:47.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:46 smithi115 ceph-mon[66621]: pgmap v128: 161 pgs: 22 active+undersized, 14 stale+active+clean, 11 active+undersized+degraded, 114 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 55/630 objects degraded (8.730%) 2024-09-20T22:44:47.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:46 smithi115 ceph-mon[66621]: Health check failed: Degraded data redundancy: 55/630 objects degraded (8.730%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:47.529 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:47 smithi153 podman[59596]: 2024-09-20 22:44:47.099695253 +0000 UTC m=+0.344122442 container init 56c41ebb736d4893fa8b0d19ed6a70af576e6cff3c6e803dfd831e33fddef536 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:44:47.529 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:47 smithi153 podman[59596]: 2024-09-20 22:44:47.103137037 +0000 UTC m=+0.347564223 container start 56c41ebb736d4893fa8b0d19ed6a70af576e6cff3c6e803dfd831e33fddef536 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:44:47.529 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:47 smithi153 podman[59596]: 2024-09-20 22:44:47.191275914 +0000 UTC m=+0.435703095 container attach 56c41ebb736d4893fa8b0d19ed6a70af576e6cff3c6e803dfd831e33fddef536 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-20T22:44:47.529 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:47 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate[59611]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:44:47.529 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:47 smithi153 bash[59596]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:44:47.529 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:47 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate[59611]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:44:47.529 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:47 smithi153 bash[59596]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:44:48.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:48 smithi153 ceph-mon[56592]: pgmap v129: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:48.899 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:48 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:48.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:44:48.900 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate[59611]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T22:44:48.900 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 bash[59596]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T22:44:48.900 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate[59611]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T22:44:48.900 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 bash[59596]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T22:44:48.900 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate[59611]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T22:44:48.900 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 bash[59596]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T22:44:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:48 smithi115 ceph-mon[64113]: pgmap v129: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:49.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:48 smithi115 ceph-mon[66621]: pgmap v129: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:49.279 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate[59611]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T22:44:49.279 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 bash[59596]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T22:44:49.279 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate[59611]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-20T22:44:49.279 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 bash[59596]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-20T22:44:49.279 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate[59611]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T22:44:49.279 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 bash[59596]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T22:44:49.279 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate[59611]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-20T22:44:49.279 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 bash[59596]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-20T22:44:49.279 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:48 smithi153 podman[59596]: 2024-09-20 22:44:48.926767614 +0000 UTC m=+2.171194798 container died 56c41ebb736d4893fa8b0d19ed6a70af576e6cff3c6e803dfd831e33fddef536 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70) 2024-09-20T22:44:49.998 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:49 smithi153 podman[59755]: 2024-09-20 22:44:49.728933752 +0000 UTC m=+0.796328947 container remove 56c41ebb736d4893fa8b0d19ed6a70af576e6cff3c6e803dfd831e33fddef536 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2) 2024-09-20T22:44:49.998 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:49 smithi153 podman[59815]: 2024-09-20 22:44:49.840989456 +0000 UTC m=+0.015798773 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:44:49.998 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:49 smithi153 podman[59815]: 2024-09-20 22:44:49.996899557 +0000 UTC m=+0.171708855 container create ba7575091ec90d25a515281807273e794daa549960a010e0ee011633a2e8a31b (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:44:50.268 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:50 smithi153 podman[59815]: 2024-09-20 22:44:50.241338866 +0000 UTC m=+0.416148172 container init ba7575091ec90d25a515281807273e794daa549960a010e0ee011633a2e8a31b (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:44:50.268 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:50 smithi153 podman[59815]: 2024-09-20 22:44:50.24462226 +0000 UTC m=+0.419431559 container start ba7575091ec90d25a515281807273e794daa549960a010e0ee011633a2e8a31b (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-20T22:44:50.529 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:50 smithi153 ceph-osd[59834]: -- 172.21.15.153:0/967960252 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555bf0ad5a40 con 0x555bf0ab3c00 2024-09-20T22:44:50.529 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:50 smithi153 bash[59815]: ba7575091ec90d25a515281807273e794daa549960a010e0ee011633a2e8a31b 2024-09-20T22:44:50.529 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:50 smithi153 systemd[1]: Started Ceph osd.4 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:44:51.023 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:50 smithi153 ceph-mon[56592]: pgmap v130: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:51.023 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:50 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:51.023 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:50 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:51.023 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:50 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:51.023 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:50 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:50 smithi115 ceph-mon[64113]: pgmap v130: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:50 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:50 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:51.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:50 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:51.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:50 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:50 smithi115 ceph-mon[66621]: pgmap v130: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:50 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:50 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:50 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:51.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:50 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:52.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:51 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:52.029 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:51 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:44:51.905+0000 7fbe48a73640 -1 Falling back to public interface 2024-09-20T22:44:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:51 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:52.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:51 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:52.887 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:52 smithi153 ceph-mon[56592]: pgmap v131: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:52 smithi115 ceph-mon[64113]: pgmap v131: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:53.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:52 smithi115 ceph-mon[66621]: pgmap v131: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:53.705 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:53 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:53.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:44:54.001 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:53 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 104/630 objects degraded (16.508%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:53 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 104/630 objects degraded (16.508%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:54.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:53 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 104/630 objects degraded (16.508%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T22:44:55.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:54 smithi153 ceph-mon[56592]: pgmap v132: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:55.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:54 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:55.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:54 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:54 smithi115 ceph-mon[64113]: pgmap v132: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:54 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:54 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:55.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:54 smithi115 ceph-mon[66621]: pgmap v132: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:55.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:54 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:55.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:54 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:55.528 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:55 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:44:55.205+0000 7fbe48a73640 -1 osd.4 0 read_superblock omap replica is missing. 2024-09-20T22:44:55.529 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:55 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:44:55.236+0000 7fbe48a73640 -1 osd.4 126 log_to_monitors true 2024-09-20T22:44:55.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:44:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:44:55] "GET /metrics HTTP/1.1" 200 38473 "" "Prometheus/2.43.0" 2024-09-20T22:44:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:55 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:55 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:55 smithi115 ceph-mon[64113]: from='osd.4 [v2:172.21.15.153:6800/1825489768,v1:172.21.15.153:6801/1825489768]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:44:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:55 smithi115 ceph-mon[64113]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:44:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:55 smithi115 ceph-mon[64113]: pgmap v133: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:55 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:55 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:55 smithi115 ceph-mon[66621]: from='osd.4 [v2:172.21.15.153:6800/1825489768,v1:172.21.15.153:6801/1825489768]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:44:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:55 smithi115 ceph-mon[66621]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:44:56.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:55 smithi115 ceph-mon[66621]: pgmap v133: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:56.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:55 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:56.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:55 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:56.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:55 smithi153 ceph-mon[56592]: from='osd.4 [v2:172.21.15.153:6800/1825489768,v1:172.21.15.153:6801/1825489768]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:44:56.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:55 smithi153 ceph-mon[56592]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:44:56.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:55 smithi153 ceph-mon[56592]: pgmap v133: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 104/630 objects degraded (16.508%) 2024-09-20T22:44:56.668 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:44:56 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:44:56.422+0000 7fbe3c816640 -1 osd.4 126 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:44:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:56 smithi153 ceph-mon[56592]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T22:44:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:56 smithi153 ceph-mon[56592]: osdmap e129: 8 total, 7 up, 8 in 2024-09-20T22:44:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:56 smithi153 ceph-mon[56592]: from='osd.4 [v2:172.21.15.153:6800/1825489768,v1:172.21.15.153:6801/1825489768]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:44:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:56 smithi153 ceph-mon[56592]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:44:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:56 smithi153 ceph-mon[56592]: from='osd.4 ' entity='osd.4' 2024-09-20T22:44:57.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:56 smithi115 ceph-mon[64113]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T22:44:57.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:56 smithi115 ceph-mon[64113]: osdmap e129: 8 total, 7 up, 8 in 2024-09-20T22:44:57.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:56 smithi115 ceph-mon[64113]: from='osd.4 [v2:172.21.15.153:6800/1825489768,v1:172.21.15.153:6801/1825489768]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:44:57.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:56 smithi115 ceph-mon[64113]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:44:57.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:56 smithi115 ceph-mon[64113]: from='osd.4 ' entity='osd.4' 2024-09-20T22:44:57.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:56 smithi115 ceph-mon[66621]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T22:44:57.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:56 smithi115 ceph-mon[66621]: osdmap e129: 8 total, 7 up, 8 in 2024-09-20T22:44:57.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:56 smithi115 ceph-mon[66621]: from='osd.4 [v2:172.21.15.153:6800/1825489768,v1:172.21.15.153:6801/1825489768]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:44:57.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:56 smithi115 ceph-mon[66621]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:44:57.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:56 smithi115 ceph-mon[66621]: from='osd.4 ' entity='osd.4' 2024-09-20T22:44:58.126 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:57 smithi153 ceph-mon[56592]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:44:58.126 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:57 smithi153 ceph-mon[56592]: osd.4 [v2:172.21.15.153:6800/1825489768,v1:172.21.15.153:6801/1825489768] boot 2024-09-20T22:44:58.126 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:57 smithi153 ceph-mon[56592]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T22:44:58.126 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:57 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:44:58.126 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:57 smithi153 ceph-mon[56592]: pgmap v136: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/630 objects degraded (16.508%) 2024-09-20T22:44:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:57 smithi115 ceph-mon[64113]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:44:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:57 smithi115 ceph-mon[64113]: osd.4 [v2:172.21.15.153:6800/1825489768,v1:172.21.15.153:6801/1825489768] boot 2024-09-20T22:44:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:57 smithi115 ceph-mon[64113]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T22:44:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:57 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:44:58.295 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:57 smithi115 ceph-mon[64113]: pgmap v136: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/630 objects degraded (16.508%) 2024-09-20T22:44:58.295 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:57 smithi115 ceph-mon[66621]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:44:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:57 smithi115 ceph-mon[66621]: osd.4 [v2:172.21.15.153:6800/1825489768,v1:172.21.15.153:6801/1825489768] boot 2024-09-20T22:44:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:57 smithi115 ceph-mon[66621]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T22:44:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:57 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:44:58.296 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:57 smithi115 ceph-mon[66621]: pgmap v136: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/630 objects degraded (16.508%) 2024-09-20T22:44:58.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:44:58 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:44:58.292+0000 7f2588e7a640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-09-20T22:44:58.982 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:58 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:58.659Z caller=alerting.go:366 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-20T22:44:58.982 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:44:58 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:44:58.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.115\", device_class=\"ssd\", hostname=\"smithi115\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.115\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:44:59.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:59 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:59.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:59 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:59.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:59 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:59.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:59 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:44:59.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:59 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:59.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:59 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:44:59.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:59 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:59.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:59 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:59.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:59 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:59.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:59 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T22:44:59.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:59 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T22:44:59.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:59 smithi153 ceph-mon[56592]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-09-20T22:44:59.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:44:59 smithi153 ceph-mon[56592]: osdmap e131: 8 total, 8 up, 8 in 2024-09-20T22:44:59.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:59.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:44:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:44:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T22:44:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T22:44:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[64113]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-09-20T22:44:59.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[64113]: osdmap e131: 8 total, 8 up, 8 in 2024-09-20T22:44:59.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:59.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:59.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:59.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:44:59.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:44:59.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:44:59.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:59.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:59.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:44:59.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T22:44:59.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T22:44:59.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[66621]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-09-20T22:44:59.656 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:44:59 smithi115 ceph-mon[66621]: osdmap e131: 8 total, 8 up, 8 in 2024-09-20T22:45:00.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:00 smithi153 ceph-mon[56592]: pgmap v138: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 104/630 objects degraded (16.508%) 2024-09-20T22:45:00.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:00 smithi115 ceph-mon[64113]: pgmap v138: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 104/630 objects degraded (16.508%) 2024-09-20T22:45:00.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:00 smithi115 ceph-mon[66621]: pgmap v138: 161 pgs: 43 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 104/630 objects degraded (16.508%) 2024-09-20T22:45:01.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:01 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:01.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:01 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:01.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:01 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:02.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:02 smithi153 ceph-mon[56592]: pgmap v139: 161 pgs: 3 active+undersized, 4 active+undersized+degraded, 154 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 16/630 objects degraded (2.540%) 2024-09-20T22:45:02.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:02 smithi115 ceph-mon[64113]: pgmap v139: 161 pgs: 3 active+undersized, 4 active+undersized+degraded, 154 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 16/630 objects degraded (2.540%) 2024-09-20T22:45:02.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:02 smithi115 ceph-mon[66621]: pgmap v139: 161 pgs: 3 active+undersized, 4 active+undersized+degraded, 154 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 16/630 objects degraded (2.540%) 2024-09-20T22:45:03.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:03 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 16/630 objects degraded (2.540%), 4 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:03.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:03 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 16/630 objects degraded (2.540%), 4 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:03.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:03 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 16/630 objects degraded (2.540%), 4 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:03.778 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:45:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:45:03.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi115\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"9fe0f662-779f-11ef-baf5-efdc52797490\", domainname=\"(none)\", instance=\"smithi115\", job=\"node\", machine=\"x86_64\", nodename=\"smithi115\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-20T22:45:04.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:04 smithi153 ceph-mon[56592]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-09-20T22:45:04.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:04 smithi115 ceph-mon[64113]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-09-20T22:45:04.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:04 smithi115 ceph-mon[66621]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-09-20T22:45:05.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:05 smithi153 ceph-mon[56592]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/630 objects degraded (2.540%), 4 pgs degraded) 2024-09-20T22:45:05.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:05 smithi153 ceph-mon[56592]: Cluster is now healthy 2024-09-20T22:45:05.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:05 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:05.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:05 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:05.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:05 smithi115 ceph-mon[64113]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/630 objects degraded (2.540%), 4 pgs degraded) 2024-09-20T22:45:05.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:05 smithi115 ceph-mon[64113]: Cluster is now healthy 2024-09-20T22:45:05.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:05 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:05.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:05 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:05.655 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:45:05 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:45:05] "GET /metrics HTTP/1.1" 200 38476 "" "Prometheus/2.43.0" 2024-09-20T22:45:05.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:05 smithi115 ceph-mon[66621]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/630 objects degraded (2.540%), 4 pgs degraded) 2024-09-20T22:45:05.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:05 smithi115 ceph-mon[66621]: Cluster is now healthy 2024-09-20T22:45:05.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:05 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:05.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:05 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:06.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:06 smithi153 ceph-mon[56592]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 619 B/s rd, 0 op/s 2024-09-20T22:45:06.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:06 smithi115 ceph-mon[64113]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 619 B/s rd, 0 op/s 2024-09-20T22:45:06.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:06 smithi115 ceph-mon[66621]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 619 B/s rd, 0 op/s 2024-09-20T22:45:09.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:08 smithi153 ceph-mon[56592]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:45:09.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:08 smithi115 ceph-mon[64113]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:45:09.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:08 smithi115 ceph-mon[66621]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:45:11.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:10 smithi153 ceph-mon[56592]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-09-20T22:45:11.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:10 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:10 smithi115 ceph-mon[64113]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-09-20T22:45:11.301 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:10 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:11.301 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:10 smithi115 ceph-mon[66621]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-09-20T22:45:11.302 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:10 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:12.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:11 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:12.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:11 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:12.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:11 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:13.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:12 smithi153 ceph-mon[56592]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:12 smithi115 ceph-mon[64113]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:13.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:12 smithi115 ceph-mon[66621]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:13.978 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:13 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T22:45:14.013 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:45:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:13 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T22:45:14.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:13 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T22:45:14.669 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:45:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5m) 54s ago 10m 26.0M - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:45:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (4m) 20s ago 9m 52.8M - dad864ee21e9 568386e3ebfd 2024-09-20T22:45:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (4m) 54s ago 9m 52.7M - 3.5 e1d6a67b021e e7840bcf555a 2024-09-20T22:45:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (4m) 20s ago 14m 440M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:45:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (4m) 54s ago 16m 507M - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:45:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4m) 54s ago 16m 46.4M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:45:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (3m) 20s ago 14m 36.7M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 f4dde7cfd242 2024-09-20T22:45:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (3m) 54s ago 15m 38.3M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 887b72a1971e 2024-09-20T22:45:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (5m) 54s ago 10m 14.7M - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:45:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (5m) 20s ago 10m 15.0M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:45:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (2m) 54s ago 14m 45.3M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 98478f94c84e 2024-09-20T22:45:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2m) 54s ago 13m 53.7M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 4c2f3b5efcd0 2024-09-20T22:45:14.670 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (94s) 54s ago 13m 50.0M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 b90ea318f712 2024-09-20T22:45:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (60s) 54s ago 13m 11.1M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 9eec1f15261b 2024-09-20T22:45:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (24s) 20s ago 12m 11.4M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 ba7575091ec9 2024-09-20T22:45:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (12m) 20s ago 12m 55.5M 2992M 17.2.0 e1d6a67b021e 6dbc08cd00a1 2024-09-20T22:45:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (12m) 20s ago 12m 53.7M 2992M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:45:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (11m) 20s ago 11m 58.4M 2992M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:45:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (4m) 20s ago 10m 64.7M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:45:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (9m) 54s ago 9m 74.9M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:45:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (9m) 20s ago 9m 75.5M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:45:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (9m) 54s ago 9m 74.1M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:45:14.671 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (9m) 20s ago 9m 71.7M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:45:15.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:14 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T22:45:15.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:14 smithi153 ceph-mon[56592]: Upgrade: osd.5 is safe to restart 2024-09-20T22:45:15.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:14 smithi153 ceph-mon[56592]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:15.070 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:45:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:45:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:45:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:45:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:45:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:45:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:45:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:45:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-09-20T22:45:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 5 2024-09-20T22:45:15.070 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:45:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:45:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:45:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:45:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:45:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-09-20T22:45:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 10 2024-09-20T22:45:15.071 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:45:15.071 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:45:15.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:14 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T22:45:15.071 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:14 smithi115 ceph-mon[64113]: Upgrade: osd.5 is safe to restart 2024-09-20T22:45:15.072 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:14 smithi115 ceph-mon[64113]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:15.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:14 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T22:45:15.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:14 smithi115 ceph-mon[66621]: Upgrade: osd.5 is safe to restart 2024-09-20T22:45:15.072 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:14 smithi115 ceph-mon[66621]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:15.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:45:15] "GET /metrics HTTP/1.1" 200 38476 "" "Prometheus/2.43.0" 2024-09-20T22:45:15.405 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:45:15.406 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:45:15.406 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:45:15.406 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:45:15.406 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:45:15.406 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr", 2024-09-20T22:45:15.406 INFO:teuthology.orchestra.run.smithi115.stdout: "mon" 2024-09-20T22:45:15.406 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:45:15.406 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "10/23 daemons upgraded", 2024-09-20T22:45:15.406 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T22:45:15.406 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:45:15.406 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:45:15.806 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:45:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[64113]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[64113]: from='client.44421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[64113]: from='client.54361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[64113]: Upgrade: Updating osd.5 2024-09-20T22:45:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T22:45:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:45:16.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[64113]: Deploying daemon osd.5 on smithi153 2024-09-20T22:45:16.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/1013639875' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[66621]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[66621]: from='client.44421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[66621]: from='client.54361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[66621]: Upgrade: Updating osd.5 2024-09-20T22:45:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T22:45:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:45:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[66621]: Deploying daemon osd.5 on smithi153 2024-09-20T22:45:16.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:15 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/1013639875' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:16.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:15 smithi153 ceph-mon[56592]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:16.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:15 smithi153 ceph-mon[56592]: from='client.44421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:16.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:15 smithi153 ceph-mon[56592]: from='client.54361 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:16.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:15 smithi153 ceph-mon[56592]: Upgrade: Updating osd.5 2024-09-20T22:45:16.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:16.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T22:45:16.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:15 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:45:16.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:15 smithi153 ceph-mon[56592]: Deploying daemon osd.5 on smithi153 2024-09-20T22:45:16.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:15 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/1013639875' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:16.907 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:16 smithi153 ceph-mon[56592]: from='client.44439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:16.907 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:16 smithi153 ceph-mon[56592]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:16.907 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:16 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/1144907309' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:45:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:16 smithi115 ceph-mon[64113]: from='client.44439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:16 smithi115 ceph-mon[64113]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:16 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/1144907309' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:45:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:16 smithi115 ceph-mon[66621]: from='client.44439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:17.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:16 smithi115 ceph-mon[66621]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:45:17.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:16 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/1144907309' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:45:18.152 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:17 smithi153 systemd[1]: Stopping Ceph osd.5 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:45:18.528 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[30837]: 2024-09-20T22:45:18.168+0000 7fe2ce6c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:45:18.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[30837]: 2024-09-20T22:45:18.168+0000 7fe2ce6c3700 -1 osd.5 131 *** Got signal Terminated *** 2024-09-20T22:45:18.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[30837]: 2024-09-20T22:45:18.168+0000 7fe2ce6c3700 -1 osd.5 131 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:45:19.094 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:18 smithi153 ceph-mon[56592]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:19.094 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:18 smithi153 ceph-mon[56592]: osd.5 marked itself down and dead 2024-09-20T22:45:19.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:18 smithi115 ceph-mon[64113]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:19.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:18 smithi115 ceph-mon[64113]: osd.5 marked itself down and dead 2024-09-20T22:45:19.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:18 smithi115 ceph-mon[66621]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:45:19.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:18 smithi115 ceph-mon[66621]: osd.5 marked itself down and dead 2024-09-20T22:45:19.367 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:19 smithi153 podman[61670]: 2024-09-20 22:45:19.136462735 +0000 UTC m=+1.060031991 container died 6dbc08cd00a1d3dc97db7c31da111698c56815fa0ba826e0d5cbc14d930482b8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_CLEAN=True, name=centos-stream, release=754, distribution-scope=public, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, vcs-type=git, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=) 2024-09-20T22:45:19.755 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:19 smithi153 podman[61670]: 2024-09-20 22:45:19.552362935 +0000 UTC m=+1.475932188 container cleanup 6dbc08cd00a1d3dc97db7c31da111698c56815fa0ba826e0d5cbc14d930482b8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , distribution-scope=public, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, GIT_BRANCH=HEAD, name=centos-stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, architecture=x86_64, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, ceph=True, com.redhat.component=centos-stream-container) 2024-09-20T22:45:19.755 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:19 smithi153 bash[61670]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5 2024-09-20T22:45:20.028 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:19 smithi153 podman[61695]: 2024-09-20 22:45:19.753982168 +0000 UTC m=+0.621054276 container remove 6dbc08cd00a1d3dc97db7c31da111698c56815fa0ba826e0d5cbc14d930482b8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., release=754, GIT_CLEAN=True, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, version=8, RELEASE=HEAD, architecture=x86_64, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, vcs-type=git, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0) 2024-09-20T22:45:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:45:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: osdmap e132: 8 total, 7 up, 8 in 2024-09-20T22:45:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]: dispatch 2024-09-20T22:45:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.7", "id": [1, 2]}]: dispatch 2024-09-20T22:45:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.11", "id": [4, 5]}]: dispatch 2024-09-20T22:45:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 5]}]: dispatch 2024-09-20T22:45:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 3]}]: dispatch 2024-09-20T22:45:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [3, 5]}]: dispatch 2024-09-20T22:45:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]: dispatch 2024-09-20T22:45:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.f", "id": [3, 5]}]: dispatch 2024-09-20T22:45:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 5]}]: dispatch 2024-09-20T22:45:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [7, 0]}]: dispatch 2024-09-20T22:45:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:20.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:45:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: osdmap e132: 8 total, 7 up, 8 in 2024-09-20T22:45:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]: dispatch 2024-09-20T22:45:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.7", "id": [1, 2]}]: dispatch 2024-09-20T22:45:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.11", "id": [4, 5]}]: dispatch 2024-09-20T22:45:20.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 5]}]: dispatch 2024-09-20T22:45:20.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 3]}]: dispatch 2024-09-20T22:45:20.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [3, 5]}]: dispatch 2024-09-20T22:45:20.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]: dispatch 2024-09-20T22:45:20.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.f", "id": [3, 5]}]: dispatch 2024-09-20T22:45:20.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 5]}]: dispatch 2024-09-20T22:45:20.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [7, 0]}]: dispatch 2024-09-20T22:45:20.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:20.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:20.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:45:20.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: osdmap e132: 8 total, 7 up, 8 in 2024-09-20T22:45:20.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]: dispatch 2024-09-20T22:45:20.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.7", "id": [1, 2]}]: dispatch 2024-09-20T22:45:20.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.11", "id": [4, 5]}]: dispatch 2024-09-20T22:45:20.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 5]}]: dispatch 2024-09-20T22:45:20.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 3]}]: dispatch 2024-09-20T22:45:20.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [3, 5]}]: dispatch 2024-09-20T22:45:20.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]: dispatch 2024-09-20T22:45:20.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.f", "id": [3, 5]}]: dispatch 2024-09-20T22:45:20.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 5]}]: dispatch 2024-09-20T22:45:20.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [7, 0]}]: dispatch 2024-09-20T22:45:20.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:20.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:20.435 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:20 smithi153 podman[61786]: 2024-09-20 22:45:20.068620651 +0000 UTC m=+0.016292334 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:45:20.435 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:20 smithi153 podman[61786]: 2024-09-20 22:45:20.205227651 +0000 UTC m=+0.152899318 container create a72b133a9b569717e20f85707206eccde34025a7a3fcb500e1fe0c1e95917729 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-09-20T22:45:20.779 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:20 smithi153 podman[61786]: 2024-09-20 22:45:20.434069892 +0000 UTC m=+0.381741575 container init a72b133a9b569717e20f85707206eccde34025a7a3fcb500e1fe0c1e95917729 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, ceph=True) 2024-09-20T22:45:20.779 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:20 smithi153 podman[61786]: 2024-09-20 22:45:20.437526343 +0000 UTC m=+0.385198011 container start a72b133a9b569717e20f85707206eccde34025a7a3fcb500e1fe0c1e95917729 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-20T22:45:20.779 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:20 smithi153 podman[61786]: 2024-09-20 22:45:20.517246536 +0000 UTC m=+0.464918205 container attach a72b133a9b569717e20f85707206eccde34025a7a3fcb500e1fe0c1e95917729 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70) 2024-09-20T22:45:20.779 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:20 smithi153 podman[61786]: 2024-09-20 22:45:20.62256082 +0000 UTC m=+0.570232488 container died a72b133a9b569717e20f85707206eccde34025a7a3fcb500e1fe0c1e95917729 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:45:21.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[64113]: pgmap v149: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:45:21.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]': finished 2024-09-20T22:45:21.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.7", "id": [1, 2]}]': finished 2024-09-20T22:45:21.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.11", "id": [4, 5]}]': finished 2024-09-20T22:45:21.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 5]}]': finished 2024-09-20T22:45:21.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 3]}]': finished 2024-09-20T22:45:21.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [3, 5]}]': finished 2024-09-20T22:45:21.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]': finished 2024-09-20T22:45:21.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.f", "id": [3, 5]}]': finished 2024-09-20T22:45:21.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 5]}]': finished 2024-09-20T22:45:21.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [7, 0]}]': finished 2024-09-20T22:45:21.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[64113]: osdmap e133: 8 total, 7 up, 8 in 2024-09-20T22:45:21.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[66621]: pgmap v149: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:45:21.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]': finished 2024-09-20T22:45:21.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.7", "id": [1, 2]}]': finished 2024-09-20T22:45:21.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.11", "id": [4, 5]}]': finished 2024-09-20T22:45:21.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 5]}]': finished 2024-09-20T22:45:21.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 3]}]': finished 2024-09-20T22:45:21.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [3, 5]}]': finished 2024-09-20T22:45:21.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]': finished 2024-09-20T22:45:21.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.f", "id": [3, 5]}]': finished 2024-09-20T22:45:21.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 5]}]': finished 2024-09-20T22:45:21.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [7, 0]}]': finished 2024-09-20T22:45:21.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:20 smithi115 ceph-mon[66621]: osdmap e133: 8 total, 7 up, 8 in 2024-09-20T22:45:21.231 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:20 smithi153 ceph-mon[56592]: pgmap v149: 161 pgs: 22 stale+active+clean, 139 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:45:21.231 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [7, 6]}]': finished 2024-09-20T22:45:21.231 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.7", "id": [1, 2]}]': finished 2024-09-20T22:45:21.231 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.11", "id": [4, 5]}]': finished 2024-09-20T22:45:21.231 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.18", "id": [4, 5]}]': finished 2024-09-20T22:45:21.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 3]}]': finished 2024-09-20T22:45:21.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.6", "id": [3, 5]}]': finished 2024-09-20T22:45:21.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]': finished 2024-09-20T22:45:21.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.f", "id": [3, 5]}]': finished 2024-09-20T22:45:21.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 5]}]': finished 2024-09-20T22:45:21.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:20 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1c", "id": [7, 0]}]': finished 2024-09-20T22:45:21.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:20 smithi153 ceph-mon[56592]: osdmap e133: 8 total, 7 up, 8 in 2024-09-20T22:45:21.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:21 smithi153 podman[61809]: 2024-09-20 22:45:21.229713568 +0000 UTC m=+0.598006389 container remove a72b133a9b569717e20f85707206eccde34025a7a3fcb500e1fe0c1e95917729 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-20T22:45:21.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:21 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.5.service: Deactivated successfully. 2024-09-20T22:45:21.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:21 smithi153 systemd[1]: Stopped Ceph osd.5 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:45:21.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:21 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.5.service: Consumed 9.012s CPU time. 2024-09-20T22:45:21.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:21 smithi153 systemd[1]: Starting Ceph osd.5 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:45:21.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:21 smithi153 ceph-mon[56592]: osdmap e134: 8 total, 7 up, 8 in 2024-09-20T22:45:21.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:21 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:21.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:21 smithi153 ceph-mon[56592]: osdmap e135: 8 total, 7 up, 8 in 2024-09-20T22:45:21.913 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:21 smithi153 podman[61908]: 2024-09-20 22:45:21.564011608 +0000 UTC m=+0.016067671 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:45:21.913 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:21 smithi153 podman[61908]: 2024-09-20 22:45:21.689281006 +0000 UTC m=+0.141337067 container create 787cc7b2be5a6c576fd2837b7f4479a75bda4f1058fe033eb2d471a38d1d8a29 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-20T22:45:21.914 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:21 smithi153 podman[61908]: 2024-09-20 22:45:21.908431026 +0000 UTC m=+0.360487094 container init 787cc7b2be5a6c576fd2837b7f4479a75bda4f1058fe033eb2d471a38d1d8a29 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate, RELEASE=squid-9e2ac70, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-09-20T22:45:21.914 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:21 smithi153 podman[61908]: 2024-09-20 22:45:21.912073113 +0000 UTC m=+0.364129183 container start 787cc7b2be5a6c576fd2837b7f4479a75bda4f1058fe033eb2d471a38d1d8a29 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-09-20T22:45:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:21 smithi115 ceph-mon[64113]: osdmap e134: 8 total, 7 up, 8 in 2024-09-20T22:45:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:21 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:21 smithi115 ceph-mon[64113]: osdmap e135: 8 total, 7 up, 8 in 2024-09-20T22:45:22.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:21 smithi115 ceph-mon[66621]: osdmap e134: 8 total, 7 up, 8 in 2024-09-20T22:45:22.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:21 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:22.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:21 smithi115 ceph-mon[66621]: osdmap e135: 8 total, 7 up, 8 in 2024-09-20T22:45:22.279 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:22 smithi153 podman[61908]: 2024-09-20 22:45:22.008513193 +0000 UTC m=+0.460569263 container attach 787cc7b2be5a6c576fd2837b7f4479a75bda4f1058fe033eb2d471a38d1d8a29 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-20T22:45:22.279 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:22 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate[61923]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:45:22.279 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:22 smithi153 bash[61908]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:45:22.279 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:22 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate[61923]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:45:22.279 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:22 smithi153 bash[61908]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:45:23.118 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:22 smithi153 ceph-mon[56592]: pgmap v152: 161 pgs: 4 remapped+peering, 2 peering, 32 active+undersized, 2 stale+active+clean, 16 active+undersized+degraded, 105 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 62/630 objects degraded (9.841%) 2024-09-20T22:45:23.118 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:22 smithi153 ceph-mon[56592]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T22:45:23.118 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:22 smithi153 ceph-mon[56592]: Health check failed: Degraded data redundancy: 62/630 objects degraded (9.841%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:23.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:22 smithi115 ceph-mon[64113]: pgmap v152: 161 pgs: 4 remapped+peering, 2 peering, 32 active+undersized, 2 stale+active+clean, 16 active+undersized+degraded, 105 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 62/630 objects degraded (9.841%) 2024-09-20T22:45:23.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:22 smithi115 ceph-mon[64113]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T22:45:23.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:22 smithi115 ceph-mon[64113]: Health check failed: Degraded data redundancy: 62/630 objects degraded (9.841%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:23.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:22 smithi115 ceph-mon[66621]: pgmap v152: 161 pgs: 4 remapped+peering, 2 peering, 32 active+undersized, 2 stale+active+clean, 16 active+undersized+degraded, 105 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 62/630 objects degraded (9.841%) 2024-09-20T22:45:23.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:22 smithi115 ceph-mon[66621]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T22:45:23.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:22 smithi115 ceph-mon[66621]: Health check failed: Degraded data redundancy: 62/630 objects degraded (9.841%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:23.405 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate[61923]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T22:45:23.405 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 bash[61908]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T22:45:23.405 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate[61923]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T22:45:23.405 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 bash[61908]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T22:45:23.405 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate[61923]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T22:45:23.740 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 bash[61908]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T22:45:23.740 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate[61923]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T22:45:23.740 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 bash[61908]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T22:45:23.740 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate[61923]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-20T22:45:23.741 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 bash[61908]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-20T22:45:23.741 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate[61923]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T22:45:23.741 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 bash[61908]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T22:45:23.741 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate[61923]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-20T22:45:23.741 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 bash[61908]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-20T22:45:23.741 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:23 smithi153 podman[61908]: 2024-09-20 22:45:23.432074737 +0000 UTC m=+1.884130802 container died 787cc7b2be5a6c576fd2837b7f4479a75bda4f1058fe033eb2d471a38d1d8a29 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-20T22:45:24.325 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:24 smithi153 podman[62051]: 2024-09-20 22:45:24.031015499 +0000 UTC m=+0.592349159 container remove 787cc7b2be5a6c576fd2837b7f4479a75bda4f1058fe033eb2d471a38d1d8a29 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-activate, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True) 2024-09-20T22:45:24.325 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:24 smithi153 podman[62112]: 2024-09-20 22:45:24.142306686 +0000 UTC m=+0.016011735 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:45:24.325 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:24 smithi153 podman[62112]: 2024-09-20 22:45:24.323775616 +0000 UTC m=+0.197480702 container create b1f12957e36e31c47c4359208b48227e1e387336f17ee860e887461d9bc5074b (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731) 2024-09-20T22:45:24.593 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:24 smithi153 podman[62112]: 2024-09-20 22:45:24.567516234 +0000 UTC m=+0.441221275 container init b1f12957e36e31c47c4359208b48227e1e387336f17ee860e887461d9bc5074b (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, ceph=True) 2024-09-20T22:45:24.593 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:24 smithi153 podman[62112]: 2024-09-20 22:45:24.570774771 +0000 UTC m=+0.444479821 container start b1f12957e36e31c47c4359208b48227e1e387336f17ee860e887461d9bc5074b (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-09-20T22:45:25.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:24 smithi153 ceph-mon[56592]: pgmap v154: 161 pgs: 7 remapped+peering, 2 peering, 35 active+undersized, 19 active+undersized+degraded, 98 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2024-09-20T22:45:25.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:24 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:25.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:24 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:25.029 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:24 smithi153 ceph-osd[62131]: -- 172.21.15.153:0/2004692202 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ccec309a40 con 0x55cced0d0400 2024-09-20T22:45:25.029 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:24 smithi153 bash[62112]: b1f12957e36e31c47c4359208b48227e1e387336f17ee860e887461d9bc5074b 2024-09-20T22:45:25.029 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:24 smithi153 systemd[1]: Started Ceph osd.5 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:45:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:24 smithi115 ceph-mon[64113]: pgmap v154: 161 pgs: 7 remapped+peering, 2 peering, 35 active+undersized, 19 active+undersized+degraded, 98 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2024-09-20T22:45:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:24 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:24 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:25.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:24 smithi115 ceph-mon[66621]: pgmap v154: 161 pgs: 7 remapped+peering, 2 peering, 35 active+undersized, 19 active+undersized+degraded, 98 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2024-09-20T22:45:25.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:24 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:25.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:24 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:25.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:45:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:45:25] "GET /metrics HTTP/1.1" 200 38476 "" "Prometheus/2.43.0" 2024-09-20T22:45:26.778 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:26 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:45:26.465+0000 7f0c2ccf3640 -1 Falling back to public interface 2024-09-20T22:45:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:26 smithi115 ceph-mon[64113]: pgmap v155: 161 pgs: 3 active+clean+remapped, 7 remapped+peering, 35 active+undersized, 19 active+undersized+degraded, 97 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2024-09-20T22:45:27.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:26 smithi115 ceph-mon[66621]: pgmap v155: 161 pgs: 3 active+clean+remapped, 7 remapped+peering, 35 active+undersized, 19 active+undersized+degraded, 97 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2024-09-20T22:45:27.183 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:26 smithi153 ceph-mon[56592]: pgmap v155: 161 pgs: 3 active+clean+remapped, 7 remapped+peering, 35 active+undersized, 19 active+undersized+degraded, 97 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2024-09-20T22:45:28.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:27 smithi153 ceph-mon[56592]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-20T22:45:28.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:27 smithi115 ceph-mon[64113]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-20T22:45:28.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:27 smithi115 ceph-mon[66621]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-20T22:45:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:28 smithi115 ceph-mon[64113]: pgmap v156: 161 pgs: 10 active+clean+remapped, 35 active+undersized, 19 active+undersized+degraded, 97 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%); 1/630 objects misplaced (0.159%); 1 B/s, 0 objects/s recovering 2024-09-20T22:45:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:28 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 71/630 objects degraded (11.270%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:28 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:28 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:28 smithi115 ceph-mon[66621]: pgmap v156: 161 pgs: 10 active+clean+remapped, 35 active+undersized, 19 active+undersized+degraded, 97 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%); 1/630 objects misplaced (0.159%); 1 B/s, 0 objects/s recovering 2024-09-20T22:45:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:28 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 71/630 objects degraded (11.270%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:28 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:28 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:29.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:28 smithi153 ceph-mon[56592]: pgmap v156: 161 pgs: 10 active+clean+remapped, 35 active+undersized, 19 active+undersized+degraded, 97 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%); 1/630 objects misplaced (0.159%); 1 B/s, 0 objects/s recovering 2024-09-20T22:45:29.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:28 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 71/630 objects degraded (11.270%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:29.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:28 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:29.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:28 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:30.009 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:45:29.759+0000 7f0c2ccf3640 -1 osd.5 0 read_superblock omap replica is missing. 2024-09-20T22:45:30.009 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:45:29.786+0000 7f0c2ccf3640 -1 osd.5 131 log_to_monitors true 2024-09-20T22:45:30.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:30 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:30.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:30 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:30.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:30 smithi115 ceph-mon[64113]: pgmap v157: 161 pgs: 10 active+clean+remapped, 35 active+undersized, 19 active+undersized+degraded, 97 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%); 1/630 objects misplaced (0.159%); 1 B/s, 0 objects/s recovering 2024-09-20T22:45:30.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:30 smithi115 ceph-mon[64113]: from='osd.5 [v2:172.21.15.153:6808/4281193171,v1:172.21.15.153:6809/4281193171]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:45:30.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:30 smithi115 ceph-mon[64113]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:45:30.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:30 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:30.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:30 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:30.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:30 smithi115 ceph-mon[66621]: pgmap v157: 161 pgs: 10 active+clean+remapped, 35 active+undersized, 19 active+undersized+degraded, 97 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%); 1/630 objects misplaced (0.159%); 1 B/s, 0 objects/s recovering 2024-09-20T22:45:30.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:30 smithi115 ceph-mon[66621]: from='osd.5 [v2:172.21.15.153:6808/4281193171,v1:172.21.15.153:6809/4281193171]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:45:30.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:30 smithi115 ceph-mon[66621]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:45:30.748 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:45:30 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:45:30.359+0000 7f0c24a9e640 -1 osd.5 131 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:45:30.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:30 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:30.749 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:30 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:30.749 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:30 smithi153 ceph-mon[56592]: pgmap v157: 161 pgs: 10 active+clean+remapped, 35 active+undersized, 19 active+undersized+degraded, 97 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%); 1/630 objects misplaced (0.159%); 1 B/s, 0 objects/s recovering 2024-09-20T22:45:30.749 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:30 smithi153 ceph-mon[56592]: from='osd.5 [v2:172.21.15.153:6808/4281193171,v1:172.21.15.153:6809/4281193171]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:45:30.749 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:30 smithi153 ceph-mon[56592]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:45:31.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:31 smithi115 ceph-mon[64113]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T22:45:31.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:31 smithi115 ceph-mon[64113]: osdmap e136: 8 total, 7 up, 8 in 2024-09-20T22:45:31.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:31 smithi115 ceph-mon[64113]: from='osd.5 [v2:172.21.15.153:6808/4281193171,v1:172.21.15.153:6809/4281193171]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:45:31.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:31 smithi115 ceph-mon[64113]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:45:31.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:31 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:31.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:31 smithi115 ceph-mon[66621]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T22:45:31.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:31 smithi115 ceph-mon[66621]: osdmap e136: 8 total, 7 up, 8 in 2024-09-20T22:45:31.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:31 smithi115 ceph-mon[66621]: from='osd.5 [v2:172.21.15.153:6808/4281193171,v1:172.21.15.153:6809/4281193171]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:45:31.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:31 smithi115 ceph-mon[66621]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:45:31.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:31 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:31.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:31 smithi153 ceph-mon[56592]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T22:45:31.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:31 smithi153 ceph-mon[56592]: osdmap e136: 8 total, 7 up, 8 in 2024-09-20T22:45:31.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:31 smithi153 ceph-mon[56592]: from='osd.5 [v2:172.21.15.153:6808/4281193171,v1:172.21.15.153:6809/4281193171]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:45:31.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:31 smithi153 ceph-mon[56592]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:45:31.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:31 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:32.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:32 smithi153 ceph-mon[56592]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:45:32.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:32 smithi153 ceph-mon[56592]: osd.5 [v2:172.21.15.153:6808/4281193171,v1:172.21.15.153:6809/4281193171] boot 2024-09-20T22:45:32.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:32 smithi153 ceph-mon[56592]: osdmap e137: 8 total, 8 up, 8 in 2024-09-20T22:45:32.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:32 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:45:32.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:32 smithi153 ceph-mon[56592]: pgmap v160: 161 pgs: 2 remapped+peering, 4 peering, 8 active+clean+remapped, 31 active+undersized, 19 active+undersized+degraded, 97 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 71/630 objects degraded (11.270%); 1 B/s, 0 objects/s recovering 2024-09-20T22:45:32.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:32 smithi153 ceph-mon[56592]: osdmap e138: 8 total, 8 up, 8 in 2024-09-20T22:45:32.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:32 smithi115 ceph-mon[64113]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:45:32.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:32 smithi115 ceph-mon[64113]: osd.5 [v2:172.21.15.153:6808/4281193171,v1:172.21.15.153:6809/4281193171] boot 2024-09-20T22:45:32.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:32 smithi115 ceph-mon[64113]: osdmap e137: 8 total, 8 up, 8 in 2024-09-20T22:45:32.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:32 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:45:32.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:32 smithi115 ceph-mon[64113]: pgmap v160: 161 pgs: 2 remapped+peering, 4 peering, 8 active+clean+remapped, 31 active+undersized, 19 active+undersized+degraded, 97 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 71/630 objects degraded (11.270%); 1 B/s, 0 objects/s recovering 2024-09-20T22:45:32.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:32 smithi115 ceph-mon[64113]: osdmap e138: 8 total, 8 up, 8 in 2024-09-20T22:45:32.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:32 smithi115 ceph-mon[66621]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:45:32.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:32 smithi115 ceph-mon[66621]: osd.5 [v2:172.21.15.153:6808/4281193171,v1:172.21.15.153:6809/4281193171] boot 2024-09-20T22:45:32.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:32 smithi115 ceph-mon[66621]: osdmap e137: 8 total, 8 up, 8 in 2024-09-20T22:45:32.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:32 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:45:32.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:32 smithi115 ceph-mon[66621]: pgmap v160: 161 pgs: 2 remapped+peering, 4 peering, 8 active+clean+remapped, 31 active+undersized, 19 active+undersized+degraded, 97 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 71/630 objects degraded (11.270%); 1 B/s, 0 objects/s recovering 2024-09-20T22:45:32.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:32 smithi115 ceph-mon[66621]: osdmap e138: 8 total, 8 up, 8 in 2024-09-20T22:45:33.154 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:45:32 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:45:32.792+0000 7f2588e7a640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:45:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:33 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:33 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:33 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:45:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:33 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:45:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:33 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:33 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:45:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:33 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:33 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:33 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:33 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T22:45:34.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:33 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T22:45:34.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:33 smithi153 ceph-mon[56592]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:45:34.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:33 smithi153 ceph-mon[56592]: osdmap e139: 8 total, 8 up, 8 in 2024-09-20T22:45:34.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:34.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:34.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:45:34.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:45:34.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:34.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:45:34.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:34.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:34.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:34.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T22:45:34.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T22:45:34.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[64113]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:45:34.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[64113]: osdmap e139: 8 total, 8 up, 8 in 2024-09-20T22:45:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:45:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:45:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:45:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T22:45:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T22:45:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[66621]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:45:34.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:33 smithi115 ceph-mon[66621]: osdmap e139: 8 total, 8 up, 8 in 2024-09-20T22:45:35.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:34 smithi153 ceph-mon[56592]: pgmap v163: 161 pgs: 5 remapped+peering, 9 peering, 5 active+clean+remapped, 29 active+undersized, 16 active+undersized+degraded, 97 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/630 objects degraded (9.841%) 2024-09-20T22:45:35.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:34 smithi115 ceph-mon[64113]: pgmap v163: 161 pgs: 5 remapped+peering, 9 peering, 5 active+clean+remapped, 29 active+undersized, 16 active+undersized+degraded, 97 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/630 objects degraded (9.841%) 2024-09-20T22:45:35.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:34 smithi115 ceph-mon[66621]: pgmap v163: 161 pgs: 5 remapped+peering, 9 peering, 5 active+clean+remapped, 29 active+undersized, 16 active+undersized+degraded, 97 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 62/630 objects degraded (9.841%) 2024-09-20T22:45:35.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:45:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:45:35] "GET /metrics HTTP/1.1" 200 38342 "" "Prometheus/2.43.0" 2024-09-20T22:45:36.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:35 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 62/630 objects degraded (9.841%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:36.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:35 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:36.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:35 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:35 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 62/630 objects degraded (9.841%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:35 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:35 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:36.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:35 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 62/630 objects degraded (9.841%), 16 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:36.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:35 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:36.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:35 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:37.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:36 smithi153 ceph-mon[56592]: pgmap v164: 161 pgs: 2 remapped+peering, 9 peering, 1 active+clean+remapped, 12 active+undersized, 5 active+undersized+degraded, 132 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 0 op/s; 18/630 objects degraded (2.857%) 2024-09-20T22:45:37.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:36 smithi115 ceph-mon[64113]: pgmap v164: 161 pgs: 2 remapped+peering, 9 peering, 1 active+clean+remapped, 12 active+undersized, 5 active+undersized+degraded, 132 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 0 op/s; 18/630 objects degraded (2.857%) 2024-09-20T22:45:37.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:36 smithi115 ceph-mon[66621]: pgmap v164: 161 pgs: 2 remapped+peering, 9 peering, 1 active+clean+remapped, 12 active+undersized, 5 active+undersized+degraded, 132 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 0 op/s; 18/630 objects degraded (2.857%) 2024-09-20T22:45:38.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:37 smithi115 ceph-mon[64113]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/630 objects degraded (2.857%), 5 pgs degraded) 2024-09-20T22:45:38.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:37 smithi115 ceph-mon[64113]: Cluster is now healthy 2024-09-20T22:45:38.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:37 smithi115 ceph-mon[66621]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/630 objects degraded (2.857%), 5 pgs degraded) 2024-09-20T22:45:38.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:37 smithi115 ceph-mon[66621]: Cluster is now healthy 2024-09-20T22:45:38.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:37 smithi153 ceph-mon[56592]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/630 objects degraded (2.857%), 5 pgs degraded) 2024-09-20T22:45:38.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:37 smithi153 ceph-mon[56592]: Cluster is now healthy 2024-09-20T22:45:39.055 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:38 smithi115 ceph-mon[64113]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 483 B/s rd, 0 op/s 2024-09-20T22:45:39.056 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:38 smithi115 ceph-mon[66621]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 483 B/s rd, 0 op/s 2024-09-20T22:45:39.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:38 smithi153 ceph-mon[56592]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 483 B/s rd, 0 op/s 2024-09-20T22:45:41.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:40 smithi115 ceph-mon[64113]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-09-20T22:45:41.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:40 smithi115 ceph-mon[66621]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-09-20T22:45:41.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:40 smithi153 ceph-mon[56592]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-09-20T22:45:42.156 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:41 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:42.157 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:41 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:42.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:41 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:43.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:42 smithi115 ceph-mon[64113]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-09-20T22:45:43.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:42 smithi115 ceph-mon[66621]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-09-20T22:45:43.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:42 smithi153 ceph-mon[56592]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 0 op/s 2024-09-20T22:45:45.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:44 smithi115 ceph-mon[64113]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2024-09-20T22:45:45.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:44 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:44 smithi115 ceph-mon[66621]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2024-09-20T22:45:45.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:44 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:45.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:44 smithi153 ceph-mon[56592]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2024-09-20T22:45:45.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:44 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:45.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:45:45 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:45:45] "GET /metrics HTTP/1.1" 200 38342 "" "Prometheus/2.43.0" 2024-09-20T22:45:46.166 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:45:46.822 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:45:46.822 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (5m) 86s ago 10m 26.0M - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:45:46.822 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (5m) 18s ago 10m 52.4M - dad864ee21e9 568386e3ebfd 2024-09-20T22:45:46.822 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (4m) 86s ago 9m 52.7M - 3.5 e1d6a67b021e e7840bcf555a 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (4m) 18s ago 15m 441M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (5m) 86s ago 16m 507M - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (4m) 86s ago 16m 46.4M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (3m) 18s ago 15m 38.5M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 f4dde7cfd242 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4m) 86s ago 15m 38.3M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 887b72a1971e 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (6m) 86s ago 11m 14.7M - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (5m) 18s ago 11m 15.3M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3m) 86s ago 14m 45.3M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 98478f94c84e 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (2m) 86s ago 14m 53.7M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 4c2f3b5efcd0 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2m) 86s ago 14m 50.0M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 b90ea318f712 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (92s) 86s ago 13m 11.1M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 9eec1f15261b 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (56s) 18s ago 13m 51.3M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 ba7575091ec9 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (22s) 18s ago 12m 11.1M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 b1f12957e36e 2024-09-20T22:45:46.823 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (12m) 18s ago 12m 54.8M 2992M 17.2.0 e1d6a67b021e e2d79aae3f55 2024-09-20T22:45:46.824 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (12m) 18s ago 12m 59.3M 2992M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:45:46.824 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (4m) 18s ago 11m 64.7M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:45:46.824 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (9m) 86s ago 9m 74.9M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:45:46.824 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (9m) 18s ago 9m 75.7M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:45:46.824 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (9m) 86s ago 9m 74.1M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:45:46.824 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (9m) 18s ago 9m 72.0M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:45:47.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:46 smithi115 ceph-mon[64113]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T22:45:47.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:46 smithi115 ceph-mon[66621]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T22:45:47.219 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:45:47.219 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:45:47.219 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:45:47.219 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:45:47.219 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:45:47.219 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:45:47.219 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:45:47.220 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:45:47.220 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-09-20T22:45:47.220 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 6 2024-09-20T22:45:47.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:45:47.220 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:45:47.220 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:45:47.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:45:47.220 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:45:47.220 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-09-20T22:45:47.220 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 11 2024-09-20T22:45:47.220 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:45:47.220 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:45:47.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:46 smithi153 ceph-mon[56592]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-20T22:45:47.550 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:45:47.550 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:45:47.550 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:45:47.550 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:45:47.550 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:45:47.551 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr", 2024-09-20T22:45:47.551 INFO:teuthology.orchestra.run.smithi115.stdout: "mon" 2024-09-20T22:45:47.551 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:45:47.551 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "11/23 daemons upgraded", 2024-09-20T22:45:47.551 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T22:45:47.551 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:45:47.551 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:45:47.951 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:45:48.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:47 smithi115 ceph-mon[64113]: from='client.54391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:48.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:47 smithi115 ceph-mon[64113]: from='client.44463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:48.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:47 smithi115 ceph-mon[64113]: from='client.44469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:48.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:47 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/2344748253' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:48.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:47 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T22:45:48.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:47 smithi115 ceph-mon[66621]: from='client.54391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:48.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:47 smithi115 ceph-mon[66621]: from='client.44463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:48.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:47 smithi115 ceph-mon[66621]: from='client.44469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:48.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:47 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/2344748253' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:48.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:47 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T22:45:48.180 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:47 smithi153 ceph-mon[56592]: from='client.54391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:48.180 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:47 smithi153 ceph-mon[56592]: from='client.44463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:48.180 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:47 smithi153 ceph-mon[56592]: from='client.44469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:48.180 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:47 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/2344748253' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:45:48.180 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:47 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T22:45:49.149 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:48 smithi153 ceph-mon[56592]: from='client.44481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:49.149 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:48 smithi153 ceph-mon[56592]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:45:49.149 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:48 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T22:45:49.149 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:48 smithi153 ceph-mon[56592]: Upgrade: osd.6 is safe to restart 2024-09-20T22:45:49.149 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:48 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/407791788' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:45:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:48 smithi115 ceph-mon[64113]: from='client.44481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:48 smithi115 ceph-mon[64113]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:45:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:48 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T22:45:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:48 smithi115 ceph-mon[64113]: Upgrade: osd.6 is safe to restart 2024-09-20T22:45:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:48 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/407791788' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:45:49.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:48 smithi115 ceph-mon[66621]: from='client.44481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:45:49.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:48 smithi115 ceph-mon[66621]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:45:49.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:48 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T22:45:49.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:48 smithi115 ceph-mon[66621]: Upgrade: osd.6 is safe to restart 2024-09-20T22:45:49.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:48 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/407791788' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:45:50.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:50 smithi153 ceph-mon[56592]: Upgrade: Updating osd.6 2024-09-20T22:45:50.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:50 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:50.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:50 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T22:45:50.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:50 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:45:50.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:50 smithi153 ceph-mon[56592]: Deploying daemon osd.6 on smithi153 2024-09-20T22:45:50.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:50 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:50.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:50 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:50.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:50 smithi153 ceph-mon[56592]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:45:50.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[64113]: Upgrade: Updating osd.6 2024-09-20T22:45:50.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:50.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T22:45:50.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:45:50.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[64113]: Deploying daemon osd.6 on smithi153 2024-09-20T22:45:50.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:50.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:50.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[64113]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:45:50.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[66621]: Upgrade: Updating osd.6 2024-09-20T22:45:50.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:50.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T22:45:50.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:45:50.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[66621]: Deploying daemon osd.6 on smithi153 2024-09-20T22:45:50.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:45:50.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:50.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:50 smithi115 ceph-mon[66621]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:45:52.007 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:51 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:51 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:52.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:51 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:52.703 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:52 smithi153 systemd[1]: Stopping Ceph osd.6 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:45:52.703 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:52 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[33719]: 2024-09-20T22:45:52.584+0000 7fd250c72700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:45:52.703 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:52 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[33719]: 2024-09-20T22:45:52.584+0000 7fd250c72700 -1 osd.6 139 *** Got signal Terminated *** 2024-09-20T22:45:52.704 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:52 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[33719]: 2024-09-20T22:45:52.584+0000 7fd250c72700 -1 osd.6 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:45:53.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:52 smithi153 ceph-mon[56592]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T22:45:53.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:52 smithi153 ceph-mon[56592]: osd.6 marked itself down and dead 2024-09-20T22:45:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:52 smithi115 ceph-mon[64113]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T22:45:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:52 smithi115 ceph-mon[64113]: osd.6 marked itself down and dead 2024-09-20T22:45:53.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:52 smithi115 ceph-mon[66621]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s 2024-09-20T22:45:53.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:52 smithi115 ceph-mon[66621]: osd.6 marked itself down and dead 2024-09-20T22:45:53.529 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:53 smithi153 podman[63965]: 2024-09-20 22:45:53.251985263 +0000 UTC m=+0.754274964 container died e2d79aae3f55ad51b65aabce0119f3b446d2f861c673f999312b46c27e9b799b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, version=8, distribution-scope=public, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git) 2024-09-20T22:45:53.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:53 smithi153 ceph-mon[56592]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:45:53.988 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:53 smithi153 ceph-mon[56592]: osdmap e140: 8 total, 7 up, 8 in 2024-09-20T22:45:53.988 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:53 smithi153 podman[63965]: 2024-09-20 22:45:53.692820943 +0000 UTC m=+1.195110635 container cleanup e2d79aae3f55ad51b65aabce0119f3b446d2f861c673f999312b46c27e9b799b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6, GIT_CLEAN=True, release=754, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, ceph=True, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=) 2024-09-20T22:45:53.988 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:53 smithi153 bash[63965]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6 2024-09-20T22:45:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:53 smithi115 ceph-mon[64113]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:45:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:53 smithi115 ceph-mon[64113]: osdmap e140: 8 total, 7 up, 8 in 2024-09-20T22:45:54.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:53 smithi115 ceph-mon[66621]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:45:54.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:53 smithi115 ceph-mon[66621]: osdmap e140: 8 total, 7 up, 8 in 2024-09-20T22:45:54.278 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:53 smithi153 podman[63982]: 2024-09-20 22:45:53.986618932 +0000 UTC m=+0.733855824 container remove e2d79aae3f55ad51b65aabce0119f3b446d2f861c673f999312b46c27e9b799b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6, version=8, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., ceph=True, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , architecture=x86_64, GIT_BRANCH=HEAD, vcs-type=git, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream) 2024-09-20T22:45:54.599 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:54 smithi153 podman[64071]: 2024-09-20 22:45:54.228298499 +0000 UTC m=+0.016493721 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:45:54.599 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:54 smithi153 podman[64071]: 2024-09-20 22:45:54.329194908 +0000 UTC m=+0.117390168 container create ee377007debe061cb1d1ca6dc58d505de2c97761f7be44aac015f2d4fa01ec7f (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:45:55.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:54 smithi153 ceph-mon[56592]: pgmap v174: 161 pgs: 14 stale+active+clean, 147 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T22:45:55.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:54 smithi153 ceph-mon[56592]: osdmap e141: 8 total, 7 up, 8 in 2024-09-20T22:45:55.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:54 smithi153 podman[64071]: 2024-09-20 22:45:54.598257561 +0000 UTC m=+0.386452777 container init ee377007debe061cb1d1ca6dc58d505de2c97761f7be44aac015f2d4fa01ec7f (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-20T22:45:55.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:54 smithi153 podman[64071]: 2024-09-20 22:45:54.601771783 +0000 UTC m=+0.389967000 container start ee377007debe061cb1d1ca6dc58d505de2c97761f7be44aac015f2d4fa01ec7f (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-20T22:45:55.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:54 smithi153 podman[64071]: 2024-09-20 22:45:54.689886664 +0000 UTC m=+0.478081890 container attach ee377007debe061cb1d1ca6dc58d505de2c97761f7be44aac015f2d4fa01ec7f (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-20T22:45:55.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:54 smithi153 podman[64071]: 2024-09-20 22:45:54.790145333 +0000 UTC m=+0.578340551 container died ee377007debe061cb1d1ca6dc58d505de2c97761f7be44aac015f2d4fa01ec7f (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-20T22:45:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:54 smithi115 ceph-mon[64113]: pgmap v174: 161 pgs: 14 stale+active+clean, 147 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T22:45:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:54 smithi115 ceph-mon[64113]: osdmap e141: 8 total, 7 up, 8 in 2024-09-20T22:45:55.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:54 smithi115 ceph-mon[66621]: pgmap v174: 161 pgs: 14 stale+active+clean, 147 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-09-20T22:45:55.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:54 smithi115 ceph-mon[66621]: osdmap e141: 8 total, 7 up, 8 in 2024-09-20T22:45:55.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:45:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:45:55] "GET /metrics HTTP/1.1" 200 38493 "" "Prometheus/2.43.0" 2024-09-20T22:45:56.014 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:55 smithi153 ceph-mon[56592]: Health check failed: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:56.015 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:55 smithi153 podman[64094]: 2024-09-20 22:45:55.56897516 +0000 UTC m=+0.772758000 container remove ee377007debe061cb1d1ca6dc58d505de2c97761f7be44aac015f2d4fa01ec7f (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-20T22:45:56.015 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:55 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.6.service: Deactivated successfully. 2024-09-20T22:45:56.015 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:55 smithi153 systemd[1]: Stopped Ceph osd.6 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:45:56.015 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:55 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.6.service: Consumed 7.526s CPU time. 2024-09-20T22:45:56.015 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:55 smithi153 systemd[1]: Starting Ceph osd.6 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:45:56.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:55 smithi115 ceph-mon[64113]: Health check failed: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:56.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:55 smithi115 ceph-mon[66621]: Health check failed: Degraded data redundancy: 29/630 objects degraded (4.603%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T22:45:56.265 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:56 smithi153 podman[64192]: 2024-09-20 22:45:55.915620835 +0000 UTC m=+0.016052876 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:45:56.265 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:56 smithi153 podman[64192]: 2024-09-20 22:45:56.020266885 +0000 UTC m=+0.120698915 container create ce882fd065276ce98de24658e7184ff829806780521c0c64aec4b1daf6cd2b8e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-20T22:45:56.529 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:56 smithi153 podman[64192]: 2024-09-20 22:45:56.264497025 +0000 UTC m=+0.364929055 container init ce882fd065276ce98de24658e7184ff829806780521c0c64aec4b1daf6cd2b8e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True) 2024-09-20T22:45:56.529 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:56 smithi153 podman[64192]: 2024-09-20 22:45:56.267990074 +0000 UTC m=+0.368422110 container start ce882fd065276ce98de24658e7184ff829806780521c0c64aec4b1daf6cd2b8e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-20T22:45:56.529 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:56 smithi153 podman[64192]: 2024-09-20 22:45:56.355948687 +0000 UTC m=+0.456380724 container attach ce882fd065276ce98de24658e7184ff829806780521c0c64aec4b1daf6cd2b8e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-20T22:45:56.529 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:56 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate[64206]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:45:56.529 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:56 smithi153 bash[64192]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:45:56.529 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:56 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate[64206]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:45:56.529 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:56 smithi153 bash[64192]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:45:57.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:56 smithi153 ceph-mon[56592]: pgmap v176: 161 pgs: 14 active+undersized, 10 stale+active+clean, 8 active+undersized+degraded, 129 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s; 29/630 objects degraded (4.603%) 2024-09-20T22:45:57.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:56 smithi115 ceph-mon[64113]: pgmap v176: 161 pgs: 14 active+undersized, 10 stale+active+clean, 8 active+undersized+degraded, 129 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s; 29/630 objects degraded (4.603%) 2024-09-20T22:45:57.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:56 smithi115 ceph-mon[66621]: pgmap v176: 161 pgs: 14 active+undersized, 10 stale+active+clean, 8 active+undersized+degraded, 129 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s; 29/630 objects degraded (4.603%) 2024-09-20T22:45:57.455 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate[64206]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T22:45:57.455 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 bash[64192]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T22:45:57.455 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate[64206]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T22:45:57.456 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 bash[64192]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T22:45:57.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate[64206]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T22:45:57.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 bash[64192]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T22:45:57.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate[64206]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T22:45:57.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 bash[64192]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T22:45:57.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate[64206]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-20T22:45:57.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 bash[64192]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-20T22:45:57.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate[64206]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T22:45:57.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 bash[64192]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T22:45:57.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate[64206]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-20T22:45:57.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 bash[64192]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-20T22:45:57.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:57 smithi153 podman[64192]: 2024-09-20 22:45:57.483387678 +0000 UTC m=+1.583819712 container died ce882fd065276ce98de24658e7184ff829806780521c0c64aec4b1daf6cd2b8e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70) 2024-09-20T22:45:58.529 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:58 smithi153 podman[64348]: 2024-09-20 22:45:58.201893178 +0000 UTC m=+0.711999675 container remove ce882fd065276ce98de24658e7184ff829806780521c0c64aec4b1daf6cd2b8e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70) 2024-09-20T22:45:58.529 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:58 smithi153 podman[64407]: 2024-09-20 22:45:58.319236839 +0000 UTC m=+0.016476869 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:45:58.529 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:58 smithi153 podman[64407]: 2024-09-20 22:45:58.435998057 +0000 UTC m=+0.133238078 container create c7f1c8d06b01a3fba233ff66d263925b7af2f6a3445a78cbce59383846e5ad08 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-20T22:45:59.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:58 smithi153 ceph-mon[56592]: pgmap v177: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 0 B/s wr, 179 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:45:59.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:58 smithi153 podman[64407]: 2024-09-20 22:45:58.66332505 +0000 UTC m=+0.360565078 container init c7f1c8d06b01a3fba233ff66d263925b7af2f6a3445a78cbce59383846e5ad08 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:45:59.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:58 smithi153 podman[64407]: 2024-09-20 22:45:58.666657918 +0000 UTC m=+0.363897939 container start c7f1c8d06b01a3fba233ff66d263925b7af2f6a3445a78cbce59383846e5ad08 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-20T22:45:59.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:58 smithi153 ceph-osd[64425]: -- 172.21.15.153:0/2786386437 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555c73b89a40 con 0x555c74950000 2024-09-20T22:45:59.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:58 smithi153 bash[64407]: c7f1c8d06b01a3fba233ff66d263925b7af2f6a3445a78cbce59383846e5ad08 2024-09-20T22:45:59.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:45:58 smithi153 systemd[1]: Started Ceph osd.6 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:45:59.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:58 smithi115 ceph-mon[64113]: pgmap v177: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 0 B/s wr, 179 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:45:59.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:58 smithi115 ceph-mon[66621]: pgmap v177: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 0 B/s wr, 179 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:59 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:00.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:45:59 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:00.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:59 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:00.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:45:59 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:00.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:59 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:00.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:45:59 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:00.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:46:00 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:46:00.317+0000 7fc5d8f21640 -1 Falling back to public interface 2024-09-20T22:46:01.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:00 smithi115 ceph-mon[64113]: pgmap v178: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:01.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:00 smithi115 ceph-mon[66621]: pgmap v178: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:01.185 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:00 smithi153 ceph-mon[56592]: pgmap v178: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 116 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:02.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:01 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:02.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:01 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:02.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:01 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:02 smithi115 ceph-mon[64113]: pgmap v179: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:03.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:02 smithi115 ceph-mon[66621]: pgmap v179: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:03.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:02 smithi153 ceph-mon[56592]: pgmap v179: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:03.867 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:46:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:46:03.611+0000 7fc5d8f21640 -1 osd.6 0 read_superblock omap replica is missing. 2024-09-20T22:46:03.867 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:46:03 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:46:03.635+0000 7fc5d8f21640 -1 osd.6 139 log_to_monitors true 2024-09-20T22:46:04.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:04.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 74/630 objects degraded (11.746%), 17 pgs degraded (PG_DEGRADED) 2024-09-20T22:46:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[64113]: from='osd.6 [v2:172.21.15.153:6816/4151520611,v1:172.21.15.153:6817/4151520611]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:46:04.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[64113]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:46:04.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:04.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:04.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 74/630 objects degraded (11.746%), 17 pgs degraded (PG_DEGRADED) 2024-09-20T22:46:04.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:04.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:04.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[66621]: from='osd.6 [v2:172.21.15.153:6816/4151520611,v1:172.21.15.153:6817/4151520611]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:46:04.156 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:03 smithi115 ceph-mon[66621]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:46:04.159 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:03 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:04.160 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:03 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:04.160 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:03 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 74/630 objects degraded (11.746%), 17 pgs degraded (PG_DEGRADED) 2024-09-20T22:46:04.160 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:03 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:04.160 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:03 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:04.160 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:03 smithi153 ceph-mon[56592]: from='osd.6 [v2:172.21.15.153:6816/4151520611,v1:172.21.15.153:6817/4151520611]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:46:04.160 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:03 smithi153 ceph-mon[56592]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:46:05.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[64113]: pgmap v180: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 83 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:05.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[64113]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T22:46:05.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[64113]: from='osd.6 [v2:172.21.15.153:6816/4151520611,v1:172.21.15.153:6817/4151520611]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:46:05.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[64113]: osdmap e142: 8 total, 7 up, 8 in 2024-09-20T22:46:05.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[64113]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:46:05.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:05.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:05.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[66621]: pgmap v180: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 83 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:05.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[66621]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T22:46:05.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[66621]: from='osd.6 [v2:172.21.15.153:6816/4151520611,v1:172.21.15.153:6817/4151520611]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:46:05.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[66621]: osdmap e142: 8 total, 7 up, 8 in 2024-09-20T22:46:05.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[66621]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:46:05.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:05.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:05.181 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:46:04 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:46:04.971+0000 7fc5cccc4640 -1 osd.6 139 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:46:05.182 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:04 smithi153 ceph-mon[56592]: pgmap v180: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 83 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:05.182 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:04 smithi153 ceph-mon[56592]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T22:46:05.182 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:04 smithi153 ceph-mon[56592]: from='osd.6 [v2:172.21.15.153:6816/4151520611,v1:172.21.15.153:6817/4151520611]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:46:05.182 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:04 smithi153 ceph-mon[56592]: osdmap e142: 8 total, 7 up, 8 in 2024-09-20T22:46:05.182 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:04 smithi153 ceph-mon[56592]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:46:05.182 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:05.182 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:05.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:46:05 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:46:05] "GET /metrics HTTP/1.1" 200 38349 "" "Prometheus/2.43.0" 2024-09-20T22:46:06.267 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:05 smithi153 ceph-mon[56592]: from='osd.6 ' entity='osd.6' 2024-09-20T22:46:06.267 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:05 smithi153 ceph-mon[56592]: pgmap v182: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:06.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:05 smithi115 ceph-mon[64113]: from='osd.6 ' entity='osd.6' 2024-09-20T22:46:06.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:05 smithi115 ceph-mon[64113]: pgmap v182: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:06.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:05 smithi115 ceph-mon[66621]: from='osd.6 ' entity='osd.6' 2024-09-20T22:46:06.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:05 smithi115 ceph-mon[66621]: pgmap v182: 161 pgs: 37 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:07.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:07 smithi153 ceph-mon[56592]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:46:07.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:07 smithi153 ceph-mon[56592]: osd.6 [v2:172.21.15.153:6816/4151520611,v1:172.21.15.153:6817/4151520611] boot 2024-09-20T22:46:07.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:07 smithi153 ceph-mon[56592]: osdmap e143: 8 total, 8 up, 8 in 2024-09-20T22:46:07.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:07 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:46:07.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:07 smithi153 ceph-mon[56592]: osdmap e144: 8 total, 8 up, 8 in 2024-09-20T22:46:07.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:46:07 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:46:07.169+0000 7f2588e7a640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:46:07.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:07 smithi115 ceph-mon[64113]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:46:07.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:07 smithi115 ceph-mon[64113]: osd.6 [v2:172.21.15.153:6816/4151520611,v1:172.21.15.153:6817/4151520611] boot 2024-09-20T22:46:07.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:07 smithi115 ceph-mon[64113]: osdmap e143: 8 total, 8 up, 8 in 2024-09-20T22:46:07.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:07 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:46:07.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:07 smithi115 ceph-mon[64113]: osdmap e144: 8 total, 8 up, 8 in 2024-09-20T22:46:07.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:07 smithi115 ceph-mon[66621]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:46:07.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:07 smithi115 ceph-mon[66621]: osd.6 [v2:172.21.15.153:6816/4151520611,v1:172.21.15.153:6817/4151520611] boot 2024-09-20T22:46:07.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:07 smithi115 ceph-mon[66621]: osdmap e143: 8 total, 8 up, 8 in 2024-09-20T22:46:07.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:07 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:46:07.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:07 smithi115 ceph-mon[66621]: osdmap e144: 8 total, 8 up, 8 in 2024-09-20T22:46:08.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:08.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:08.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:08.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:46:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:46:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T22:46:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T22:46:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[64113]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:46:08.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[64113]: pgmap v185: 161 pgs: 1 peering, 36 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 0 B/s wr, 235 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:08.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:46:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:46:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T22:46:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T22:46:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[66621]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:46:08.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:08 smithi115 ceph-mon[66621]: pgmap v185: 161 pgs: 1 peering, 36 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 0 B/s wr, 235 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:08 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:08 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:08 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:08 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:46:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:08 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:08 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:46:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:08 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:08 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:08 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:08 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T22:46:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:08 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T22:46:08.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:08 smithi153 ceph-mon[56592]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-20T22:46:08.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:08 smithi153 ceph-mon[56592]: pgmap v185: 161 pgs: 1 peering, 36 active+undersized, 17 active+undersized+degraded, 107 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 0 B/s wr, 235 op/s; 74/630 objects degraded (11.746%) 2024-09-20T22:46:10.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:09 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 62/630 objects degraded (9.841%), 14 pgs degraded (PG_DEGRADED) 2024-09-20T22:46:10.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:09 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 62/630 objects degraded (9.841%), 14 pgs degraded (PG_DEGRADED) 2024-09-20T22:46:10.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:09 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 62/630 objects degraded (9.841%), 14 pgs degraded (PG_DEGRADED) 2024-09-20T22:46:11.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:10 smithi115 ceph-mon[64113]: pgmap v186: 161 pgs: 1 peering, 25 active+undersized, 14 active+undersized+degraded, 121 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 0 B/s wr, 235 op/s; 62/630 objects degraded (9.841%); 0 B/s, 1 objects/s recovering 2024-09-20T22:46:11.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:10 smithi115 ceph-mon[66621]: pgmap v186: 161 pgs: 1 peering, 25 active+undersized, 14 active+undersized+degraded, 121 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 0 B/s wr, 235 op/s; 62/630 objects degraded (9.841%); 0 B/s, 1 objects/s recovering 2024-09-20T22:46:11.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:10 smithi153 ceph-mon[56592]: pgmap v186: 161 pgs: 1 peering, 25 active+undersized, 14 active+undersized+degraded, 121 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 0 B/s wr, 235 op/s; 62/630 objects degraded (9.841%); 0 B/s, 1 objects/s recovering 2024-09-20T22:46:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:11 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:12.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:11 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:11 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:12 smithi115 ceph-mon[64113]: pgmap v187: 161 pgs: 1 peering, 160 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 0 B/s wr, 196 op/s; 0 B/s, 7 objects/s recovering 2024-09-20T22:46:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:12 smithi115 ceph-mon[64113]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/630 objects degraded (9.841%), 14 pgs degraded) 2024-09-20T22:46:13.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:12 smithi115 ceph-mon[64113]: Cluster is now healthy 2024-09-20T22:46:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:12 smithi115 ceph-mon[66621]: pgmap v187: 161 pgs: 1 peering, 160 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 0 B/s wr, 196 op/s; 0 B/s, 7 objects/s recovering 2024-09-20T22:46:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:12 smithi115 ceph-mon[66621]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/630 objects degraded (9.841%), 14 pgs degraded) 2024-09-20T22:46:13.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:12 smithi115 ceph-mon[66621]: Cluster is now healthy 2024-09-20T22:46:13.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:12 smithi153 ceph-mon[56592]: pgmap v187: 161 pgs: 1 peering, 160 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 0 B/s wr, 196 op/s; 0 B/s, 7 objects/s recovering 2024-09-20T22:46:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:12 smithi153 ceph-mon[56592]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/630 objects degraded (9.841%), 14 pgs degraded) 2024-09-20T22:46:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:12 smithi153 ceph-mon[56592]: Cluster is now healthy 2024-09-20T22:46:15.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:14 smithi115 ceph-mon[64113]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s; 0 B/s, 6 objects/s recovering 2024-09-20T22:46:15.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:14 smithi115 ceph-mon[66621]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s; 0 B/s, 6 objects/s recovering 2024-09-20T22:46:15.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:14 smithi153 ceph-mon[56592]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 113 op/s; 0 B/s, 6 objects/s recovering 2024-09-20T22:46:15.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:46:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:46:15] "GET /metrics HTTP/1.1" 200 38349 "" "Prometheus/2.43.0" 2024-09-20T22:46:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:16 smithi115 ceph-mon[64113]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T22:46:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:16 smithi115 ceph-mon[66621]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T22:46:17.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:16 smithi153 ceph-mon[56592]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T22:46:18.316 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:46:18.971 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:46:18.971 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (6m) 119s ago 11m 26.0M - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:46:18.971 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (6m) 16s ago 10m 52.7M - dad864ee21e9 568386e3ebfd 2024-09-20T22:46:18.971 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (5m) 119s ago 10m 52.7M - 3.5 e1d6a67b021e e7840bcf555a 2024-09-20T22:46:18.971 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (5m) 16s ago 15m 441M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:46:18.971 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (5m) 119s ago 17m 507M - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:46:18.971 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5m) 119s ago 17m 46.4M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:46:18.971 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (4m) 16s ago 16m 40.0M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 f4dde7cfd242 2024-09-20T22:46:18.971 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (4m) 119s ago 16m 38.3M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 887b72a1971e 2024-09-20T22:46:18.971 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (6m) 119s ago 11m 14.7M - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:46:18.971 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (6m) 16s ago 11m 15.3M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:46:18.971 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (3m) 119s ago 15m 45.3M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 98478f94c84e 2024-09-20T22:46:18.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3m) 119s ago 15m 53.7M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 4c2f3b5efcd0 2024-09-20T22:46:18.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (2m) 119s ago 14m 50.0M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 b90ea318f712 2024-09-20T22:46:18.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2m) 119s ago 14m 11.1M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 9eec1f15261b 2024-09-20T22:46:18.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (89s) 16s ago 13m 55.1M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 ba7575091ec9 2024-09-20T22:46:18.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (54s) 16s ago 13m 44.5M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 b1f12957e36e 2024-09-20T22:46:18.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (20s) 16s ago 13m 12.1M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 c7f1c8d06b01 2024-09-20T22:46:18.972 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (12m) 16s ago 12m 61.8M 2992M 17.2.0 e1d6a67b021e b2f8f36a4a4c 2024-09-20T22:46:18.972 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (5m) 16s ago 12m 69.0M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:46:18.972 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (10m) 119s ago 10m 74.9M - 17.2.0 e1d6a67b021e eee750b0891b 2024-09-20T22:46:18.972 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (10m) 16s ago 10m 76.0M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:46:18.972 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (10m) 119s ago 10m 74.1M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:46:18.972 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (10m) 16s ago 10m 72.2M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:46:19.096 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:18 smithi153 ceph-mon[56592]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T22:46:19.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:18 smithi115 ceph-mon[64113]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T22:46:19.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:18 smithi115 ceph-mon[66621]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 0 B/s, 5 objects/s recovering 2024-09-20T22:46:19.381 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:46:19.381 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:46:19.381 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:46:19.382 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:46:19.382 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:46:19.382 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:46:19.382 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:46:19.382 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:46:19.382 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-09-20T22:46:19.382 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 7 2024-09-20T22:46:19.382 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:46:19.382 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:46:19.382 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-20T22:46:19.382 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:46:19.382 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:46:19.382 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-09-20T22:46:19.383 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 12 2024-09-20T22:46:19.383 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:46:19.383 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:46:19.716 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:46:19.716 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:46:19.716 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:46:19.716 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:46:19.716 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:46:19.717 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr", 2024-09-20T22:46:19.717 INFO:teuthology.orchestra.run.smithi115.stdout: "mon" 2024-09-20T22:46:19.717 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:46:19.717 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "12/23 daemons upgraded", 2024-09-20T22:46:19.717 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T22:46:19.717 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:46:19.717 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:46:20.129 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:46:20.129 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:19 smithi115 ceph-mon[66621]: from='client.44496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:20.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:19 smithi115 ceph-mon[66621]: from='client.54436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:20.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:19 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/2293000087' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:20.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:20.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:20.130 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:20.130 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:19 smithi115 ceph-mon[64113]: from='client.44496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:20.131 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:19 smithi115 ceph-mon[64113]: from='client.54436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:20.131 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:19 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/2293000087' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:20.131 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:20.131 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:20.131 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:20.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:19 smithi153 ceph-mon[56592]: from='client.44496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:20.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:19 smithi153 ceph-mon[56592]: from='client.54436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:20.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:19 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/2293000087' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:20.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:20.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:20.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:21.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:20 smithi115 ceph-mon[64113]: from='client.44508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:21.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:20 smithi115 ceph-mon[64113]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s; 0 B/s, 4 objects/s recovering 2024-09-20T22:46:21.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:20 smithi115 ceph-mon[64113]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:21.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:20 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/2141546311' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:46:21.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:20 smithi115 ceph-mon[66621]: from='client.44508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:21.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:20 smithi115 ceph-mon[66621]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s; 0 B/s, 4 objects/s recovering 2024-09-20T22:46:21.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:20 smithi115 ceph-mon[66621]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:21.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:20 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/2141546311' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:46:21.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:20 smithi153 ceph-mon[56592]: from='client.44508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:21.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:20 smithi153 ceph-mon[56592]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s; 0 B/s, 4 objects/s recovering 2024-09-20T22:46:21.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:20 smithi153 ceph-mon[56592]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:21.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:20 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/2141546311' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:46:23.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:22 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:23.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:22 smithi115 ceph-mon[64113]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 3 objects/s recovering 2024-09-20T22:46:23.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:22 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T22:46:23.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:22 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:23.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:22 smithi115 ceph-mon[66621]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 3 objects/s recovering 2024-09-20T22:46:23.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:22 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T22:46:23.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:22 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:23.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:22 smithi153 ceph-mon[56592]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 0 B/s, 3 objects/s recovering 2024-09-20T22:46:23.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:22 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T22:46:24.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:23 smithi153 ceph-mon[56592]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T22:46:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:23 smithi153 ceph-mon[56592]: Upgrade: osd.7 is safe to restart 2024-09-20T22:46:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:23 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:23 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T22:46:24.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:23 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:24.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:23 smithi115 ceph-mon[64113]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T22:46:24.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:23 smithi115 ceph-mon[64113]: Upgrade: osd.7 is safe to restart 2024-09-20T22:46:24.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:23 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:24.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:23 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T22:46:24.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:23 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:24.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:23 smithi115 ceph-mon[66621]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T22:46:24.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:23 smithi115 ceph-mon[66621]: Upgrade: osd.7 is safe to restart 2024-09-20T22:46:24.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:23 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:24.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:23 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T22:46:24.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:23 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:25.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:24 smithi153 ceph-mon[56592]: Upgrade: Updating osd.7 2024-09-20T22:46:25.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:24 smithi153 ceph-mon[56592]: Deploying daemon osd.7 on smithi153 2024-09-20T22:46:25.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:24 smithi153 ceph-mon[56592]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:24 smithi115 ceph-mon[64113]: Upgrade: Updating osd.7 2024-09-20T22:46:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:24 smithi115 ceph-mon[64113]: Deploying daemon osd.7 on smithi153 2024-09-20T22:46:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:24 smithi115 ceph-mon[64113]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:25.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:24 smithi115 ceph-mon[66621]: Upgrade: Updating osd.7 2024-09-20T22:46:25.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:24 smithi115 ceph-mon[66621]: Deploying daemon osd.7 on smithi153 2024-09-20T22:46:25.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:24 smithi115 ceph-mon[66621]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:25.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:46:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:46:25] "GET /metrics HTTP/1.1" 200 38504 "" "Prometheus/2.43.0" 2024-09-20T22:46:26.998 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:26 smithi153 ceph-mon[56592]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:26.999 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:26 smithi153 systemd[1]: Stopping Ceph osd.7 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:46:27.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:26 smithi115 ceph-mon[64113]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:27.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:26 smithi115 ceph-mon[66621]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:46:27.278 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:26 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[36617]: 2024-09-20T22:46:26.996+0000 7fbdd36fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:46:27.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:26 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[36617]: 2024-09-20T22:46:26.996+0000 7fbdd36fa700 -1 osd.7 144 *** Got signal Terminated *** 2024-09-20T22:46:27.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:26 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[36617]: 2024-09-20T22:46:26.996+0000 7fbdd36fa700 -1 osd.7 144 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:46:28.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:27 smithi115 ceph-mon[64113]: osd.7 marked itself down and dead 2024-09-20T22:46:28.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:27 smithi115 ceph-mon[66621]: osd.7 marked itself down and dead 2024-09-20T22:46:28.183 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:27 smithi153 ceph-mon[56592]: osd.7 marked itself down and dead 2024-09-20T22:46:28.482 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:28 smithi153 podman[66275]: 2024-09-20 22:46:28.182677824 +0000 UTC m=+1.310911976 container died b2f8f36a4a4cf31d0fb8b13a7fd23515e5b5e326ac50291db321394310ccddd5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-09-20T22:46:28.778 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:28 smithi153 podman[66275]: 2024-09-20 22:46:28.646043169 +0000 UTC m=+1.774277351 container cleanup b2f8f36a4a4cf31d0fb8b13a7fd23515e5b5e326ac50291db321394310ccddd5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, version=8, vcs-type=git, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, distribution-scope=public, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-09-20T22:46:28.779 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:28 smithi153 bash[66275]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7 2024-09-20T22:46:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:28 smithi115 ceph-mon[64113]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:28 smithi115 ceph-mon[64113]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:46:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:28 smithi115 ceph-mon[64113]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-20T22:46:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:28 smithi115 ceph-mon[64113]: osdmap e145: 8 total, 7 up, 8 in 2024-09-20T22:46:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:28 smithi115 ceph-mon[66621]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:28 smithi115 ceph-mon[66621]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:46:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:28 smithi115 ceph-mon[66621]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-20T22:46:29.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:28 smithi115 ceph-mon[66621]: osdmap e145: 8 total, 7 up, 8 in 2024-09-20T22:46:29.278 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:28 smithi153 podman[66290]: 2024-09-20 22:46:28.956053236 +0000 UTC m=+0.768368255 container remove b2f8f36a4a4cf31d0fb8b13a7fd23515e5b5e326ac50291db321394310ccddd5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, ceph=True, io.buildah.version=1.19.8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux ) 2024-09-20T22:46:29.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:28 smithi153 ceph-mon[56592]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:46:29.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:28 smithi153 ceph-mon[56592]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:46:29.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:28 smithi153 ceph-mon[56592]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-20T22:46:29.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:28 smithi153 ceph-mon[56592]: osdmap e145: 8 total, 7 up, 8 in 2024-09-20T22:46:29.685 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:29 smithi153 podman[66379]: 2024-09-20 22:46:29.3238078 +0000 UTC m=+0.016005269 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:46:29.685 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:29 smithi153 podman[66379]: 2024-09-20 22:46:29.44862425 +0000 UTC m=+0.140821748 container create f6e4bc7c2c21e504248303dd35b01c64220c63574ce11c7f404faf0b908e4147 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:46:30.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:29 smithi153 ceph-mon[56592]: osdmap e146: 8 total, 7 up, 8 in 2024-09-20T22:46:30.029 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:29 smithi153 podman[66379]: 2024-09-20 22:46:29.684401583 +0000 UTC m=+0.376599044 container init f6e4bc7c2c21e504248303dd35b01c64220c63574ce11c7f404faf0b908e4147 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:46:30.029 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:29 smithi153 podman[66379]: 2024-09-20 22:46:29.68776983 +0000 UTC m=+0.379967295 container start f6e4bc7c2c21e504248303dd35b01c64220c63574ce11c7f404faf0b908e4147 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:46:30.029 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:29 smithi153 podman[66379]: 2024-09-20 22:46:29.77600761 +0000 UTC m=+0.468205078 container attach f6e4bc7c2c21e504248303dd35b01c64220c63574ce11c7f404faf0b908e4147 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-20T22:46:30.029 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:29 smithi153 podman[66379]: 2024-09-20 22:46:29.874892873 +0000 UTC m=+0.567090329 container died f6e4bc7c2c21e504248303dd35b01c64220c63574ce11c7f404faf0b908e4147 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-20T22:46:30.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:29 smithi115 ceph-mon[64113]: osdmap e146: 8 total, 7 up, 8 in 2024-09-20T22:46:30.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:29 smithi115 ceph-mon[66621]: osdmap e146: 8 total, 7 up, 8 in 2024-09-20T22:46:30.779 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:30 smithi153 podman[66402]: 2024-09-20 22:46:30.47109952 +0000 UTC m=+0.586205410 container remove f6e4bc7c2c21e504248303dd35b01c64220c63574ce11c7f404faf0b908e4147 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:46:30.779 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:30 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.7.service: Deactivated successfully. 2024-09-20T22:46:30.779 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:30 smithi153 systemd[1]: Stopped Ceph osd.7 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:46:30.779 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:30 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.7.service: Consumed 10.828s CPU time. 2024-09-20T22:46:30.779 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:30 smithi153 systemd[1]: Starting Ceph osd.7 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:46:31.126 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:30 smithi153 ceph-mon[56592]: pgmap v198: 161 pgs: 10 active+undersized, 16 stale+active+clean, 1 active+undersized+degraded, 134 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 1/630 objects degraded (0.159%) 2024-09-20T22:46:31.126 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:30 smithi153 ceph-mon[56592]: Health check failed: Degraded data redundancy: 1/630 objects degraded (0.159%), 1 pg degraded (PG_DEGRADED) 2024-09-20T22:46:31.127 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:30 smithi153 podman[66502]: 2024-09-20 22:46:30.805506019 +0000 UTC m=+0.016066832 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:46:31.127 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:30 smithi153 podman[66502]: 2024-09-20 22:46:30.922412756 +0000 UTC m=+0.132973553 container create 124ac43001cc741287be8ad8419cc7372c73d87fbe3d2cee6b4cfaca92ae65dc (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-20T22:46:31.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:30 smithi115 ceph-mon[64113]: pgmap v198: 161 pgs: 10 active+undersized, 16 stale+active+clean, 1 active+undersized+degraded, 134 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 1/630 objects degraded (0.159%) 2024-09-20T22:46:31.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:30 smithi115 ceph-mon[64113]: Health check failed: Degraded data redundancy: 1/630 objects degraded (0.159%), 1 pg degraded (PG_DEGRADED) 2024-09-20T22:46:31.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:30 smithi115 ceph-mon[66621]: pgmap v198: 161 pgs: 10 active+undersized, 16 stale+active+clean, 1 active+undersized+degraded, 134 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 1/630 objects degraded (0.159%) 2024-09-20T22:46:31.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:30 smithi115 ceph-mon[66621]: Health check failed: Degraded data redundancy: 1/630 objects degraded (0.159%), 1 pg degraded (PG_DEGRADED) 2024-09-20T22:46:31.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:31 smithi153 podman[66502]: 2024-09-20 22:46:31.125192617 +0000 UTC m=+0.335753431 container init 124ac43001cc741287be8ad8419cc7372c73d87fbe3d2cee6b4cfaca92ae65dc (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:46:31.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:31 smithi153 podman[66502]: 2024-09-20 22:46:31.129100784 +0000 UTC m=+0.339661583 container start 124ac43001cc741287be8ad8419cc7372c73d87fbe3d2cee6b4cfaca92ae65dc (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate, ceph=True, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:46:31.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:31 smithi153 podman[66502]: 2024-09-20 22:46:31.224739152 +0000 UTC m=+0.435299958 container attach 124ac43001cc741287be8ad8419cc7372c73d87fbe3d2cee6b4cfaca92ae65dc (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-20T22:46:31.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:31 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate[66516]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:46:31.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:31 smithi153 bash[66502]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:46:31.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:31 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate[66516]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:46:31.530 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:31 smithi153 bash[66502]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:46:32.009 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:31 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate[66516]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T22:46:32.009 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:31 smithi153 bash[66502]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T22:46:32.009 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:31 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate[66516]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T22:46:32.009 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:31 smithi153 bash[66502]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T22:46:32.010 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate[66516]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T22:46:32.010 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 bash[66502]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T22:46:32.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate[66516]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T22:46:32.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 bash[66502]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T22:46:32.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate[66516]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-20T22:46:32.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 bash[66502]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-20T22:46:32.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate[66516]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T22:46:32.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 bash[66502]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T22:46:32.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate[66516]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-20T22:46:32.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 bash[66502]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-20T22:46:32.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 podman[66502]: 2024-09-20 22:46:32.036672071 +0000 UTC m=+1.247232870 container died 124ac43001cc741287be8ad8419cc7372c73d87fbe3d2cee6b4cfaca92ae65dc (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-20T22:46:32.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:32 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:32.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:32 smithi153 ceph-mon[56592]: pgmap v199: 161 pgs: 32 active+undersized, 3 stale+active+clean, 22 active+undersized+degraded, 104 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2024-09-20T22:46:32.872 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 podman[66642]: 2024-09-20 22:46:32.620131508 +0000 UTC m=+0.576580181 container remove 124ac43001cc741287be8ad8419cc7372c73d87fbe3d2cee6b4cfaca92ae65dc (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-activate, RELEASE=squid-9e2ac70, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-20T22:46:32.872 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 podman[66700]: 2024-09-20 22:46:32.728993216 +0000 UTC m=+0.016107147 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:46:33.141 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:32 smithi153 podman[66700]: 2024-09-20 22:46:32.871145163 +0000 UTC m=+0.158259077 container create ca686c937f4206cbb65def3b2f4acfc04742e0e391f65ae08ab4ffdfd0c21026 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70) 2024-09-20T22:46:33.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:32 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:33.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:32 smithi115 ceph-mon[64113]: pgmap v199: 161 pgs: 32 active+undersized, 3 stale+active+clean, 22 active+undersized+degraded, 104 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2024-09-20T22:46:33.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:32 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:33.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:32 smithi115 ceph-mon[66621]: pgmap v199: 161 pgs: 32 active+undersized, 3 stale+active+clean, 22 active+undersized+degraded, 104 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2024-09-20T22:46:33.528 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:33 smithi153 podman[66700]: 2024-09-20 22:46:33.139897313 +0000 UTC m=+0.427011234 container init ca686c937f4206cbb65def3b2f4acfc04742e0e391f65ae08ab4ffdfd0c21026 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:46:33.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:33 smithi153 podman[66700]: 2024-09-20 22:46:33.143209727 +0000 UTC m=+0.430323639 container start ca686c937f4206cbb65def3b2f4acfc04742e0e391f65ae08ab4ffdfd0c21026 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:46:33.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:33 smithi153 ceph-osd[66719]: -- 172.21.15.153:0/3062984179 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555ec864da40 con 0x555ec9414400 2024-09-20T22:46:33.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:33 smithi153 bash[66700]: ca686c937f4206cbb65def3b2f4acfc04742e0e391f65ae08ab4ffdfd0c21026 2024-09-20T22:46:33.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:33 smithi153 systemd[1]: Started Ceph osd.7 for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:46:34.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:34 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:34.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:34 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:34.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:34 smithi115 ceph-mon[64113]: pgmap v200: 161 pgs: 40 active+undersized, 27 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-09-20T22:46:34.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:34 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:34.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:34 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:34.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:34 smithi115 ceph-mon[66621]: pgmap v200: 161 pgs: 40 active+undersized, 27 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-09-20T22:46:34.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:34 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:34.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:34 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:34.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:34 smithi153 ceph-mon[56592]: pgmap v200: 161 pgs: 40 active+undersized, 27 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-09-20T22:46:35.278 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:34 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:46:34.793+0000 7f859ce1a640 -1 Falling back to public interface 2024-09-20T22:46:35.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:46:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:46:35] "GET /metrics HTTP/1.1" 200 38589 "" "Prometheus/2.43.0" 2024-09-20T22:46:36.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:35 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:36.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:35 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:35 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:35 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:36.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:35 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:36.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:35 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:36.973 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:36 smithi153 ceph-mon[56592]: pgmap v201: 161 pgs: 40 active+undersized, 27 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-09-20T22:46:37.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:36 smithi115 ceph-mon[64113]: pgmap v201: 161 pgs: 40 active+undersized, 27 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-09-20T22:46:37.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:36 smithi115 ceph-mon[66621]: pgmap v201: 161 pgs: 40 active+undersized, 27 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-09-20T22:46:38.414 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:46:38.091+0000 7f859ce1a640 -1 osd.7 0 read_superblock omap replica is missing. 2024-09-20T22:46:38.414 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:46:38.126+0000 7f859ce1a640 -1 osd.7 144 log_to_monitors true 2024-09-20T22:46:38.414 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:38 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:38.414 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:38 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:38.414 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:38 smithi153 ceph-mon[56592]: pgmap v202: 161 pgs: 40 active+undersized, 27 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-09-20T22:46:38.414 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:38 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:38.415 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:38 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:38.415 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:38 smithi153 ceph-mon[56592]: from='osd.7 [v2:172.21.15.153:6824/2019437120,v1:172.21.15.153:6825/2019437120]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:46:38.415 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:38 smithi153 ceph-mon[56592]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:46:38.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:38.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:38.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[66621]: pgmap v202: 161 pgs: 40 active+undersized, 27 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-09-20T22:46:38.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:38.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:38.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[66621]: from='osd.7 [v2:172.21.15.153:6824/2019437120,v1:172.21.15.153:6825/2019437120]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:46:38.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[66621]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:46:38.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:38.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:38.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[64113]: pgmap v202: 161 pgs: 40 active+undersized, 27 active+undersized+degraded, 94 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-09-20T22:46:38.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:38.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:38.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[64113]: from='osd.7 [v2:172.21.15.153:6824/2019437120,v1:172.21.15.153:6825/2019437120]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:46:38.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:38 smithi115 ceph-mon[64113]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:46:39.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:39 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 83/630 objects degraded (13.175%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T22:46:39.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:39 smithi153 ceph-mon[56592]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T22:46:39.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:39 smithi153 ceph-mon[56592]: from='osd.7 [v2:172.21.15.153:6824/2019437120,v1:172.21.15.153:6825/2019437120]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:46:39.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:39 smithi153 ceph-mon[56592]: osdmap e147: 8 total, 7 up, 8 in 2024-09-20T22:46:39.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:39 smithi153 ceph-mon[56592]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:46:39.504 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:46:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:46:39.348+0000 7f8590bbd640 -1 osd.7 144 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:46:39.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:39 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 83/630 objects degraded (13.175%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T22:46:39.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:39 smithi115 ceph-mon[64113]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T22:46:39.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:39 smithi115 ceph-mon[64113]: from='osd.7 [v2:172.21.15.153:6824/2019437120,v1:172.21.15.153:6825/2019437120]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:46:39.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:39 smithi115 ceph-mon[64113]: osdmap e147: 8 total, 7 up, 8 in 2024-09-20T22:46:39.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:39 smithi115 ceph-mon[64113]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:46:39.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:39 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 83/630 objects degraded (13.175%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T22:46:39.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:39 smithi115 ceph-mon[66621]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T22:46:39.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:39 smithi115 ceph-mon[66621]: from='osd.7 [v2:172.21.15.153:6824/2019437120,v1:172.21.15.153:6825/2019437120]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:46:39.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:39 smithi115 ceph-mon[66621]: osdmap e147: 8 total, 7 up, 8 in 2024-09-20T22:46:39.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:39 smithi115 ceph-mon[66621]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-09-20T22:46:40.629 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:40 smithi153 ceph-mon[56592]: from='osd.7 ' entity='osd.7' 2024-09-20T22:46:40.629 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:40 smithi153 ceph-mon[56592]: pgmap v204: 161 pgs: 40 active+undersized, 27 active+undersized+degraded, 94 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-09-20T22:46:40.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:40 smithi115 ceph-mon[66621]: from='osd.7 ' entity='osd.7' 2024-09-20T22:46:40.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:40 smithi115 ceph-mon[66621]: pgmap v204: 161 pgs: 40 active+undersized, 27 active+undersized+degraded, 94 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-09-20T22:46:40.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:40 smithi115 ceph-mon[64113]: from='osd.7 ' entity='osd.7' 2024-09-20T22:46:40.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:40 smithi115 ceph-mon[64113]: pgmap v204: 161 pgs: 40 active+undersized, 27 active+undersized+degraded, 94 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-09-20T22:46:41.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:46:41.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: osd.7 [v2:172.21.15.153:6824/2019437120,v1:172.21.15.153:6825/2019437120] boot 2024-09-20T22:46:41.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: osdmap e148: 8 total, 8 up, 8 in 2024-09-20T22:46:41.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:46:41.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:41.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:41.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:41.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:46:41.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:41.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-20T22:46:41.656 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-20T22:46:41.657 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-20T22:46:41.657 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-20T22:46:41.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:46:41.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: osd.7 [v2:172.21.15.153:6824/2019437120,v1:172.21.15.153:6825/2019437120] boot 2024-09-20T22:46:41.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: osdmap e148: 8 total, 8 up, 8 in 2024-09-20T22:46:41.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:46:41.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:41.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:41.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:41.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:46:41.657 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:41.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:46:41.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:41.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:41.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:41.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:41.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:41.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-20T22:46:41.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-20T22:46:41.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-20T22:46:41.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-20T22:46:41.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-20T22:46:41.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-20T22:46:41.658 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-20T22:46:41.659 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-20T22:46:41.659 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-20T22:46:41.659 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-20T22:46:41.659 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-20T22:46:41.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-20T22:46:41.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-20T22:46:41.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-20T22:46:41.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-20T22:46:41.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-20T22:46:41.659 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-09-20T22:46:41.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-20T22:46:41.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-20T22:46:41.660 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[64113]: osdmap e149: 8 total, 8 up, 8 in 2024-09-20T22:46:41.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-20T22:46:41.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-20T22:46:41.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-20T22:46:41.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-20T22:46:41.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-20T22:46:41.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-09-20T22:46:41.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-20T22:46:41.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-20T22:46:41.660 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:41 smithi115 ceph-mon[66621]: osdmap e149: 8 total, 8 up, 8 in 2024-09-20T22:46:41.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:46:41.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: osd.7 [v2:172.21.15.153:6824/2019437120,v1:172.21.15.153:6825/2019437120] boot 2024-09-20T22:46:41.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: osdmap e148: 8 total, 8 up, 8 in 2024-09-20T22:46:41.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:46:41.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:41.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-20T22:46:41.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-20T22:46:41.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-20T22:46:41.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-20T22:46:41.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-20T22:46:41.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-20T22:46:41.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-20T22:46:41.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-20T22:46:41.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-20T22:46:41.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-20T22:46:41.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-20T22:46:41.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-09-20T22:46:41.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-20T22:46:41.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-20T22:46:41.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:41 smithi153 ceph-mon[56592]: osdmap e149: 8 total, 8 up, 8 in 2024-09-20T22:46:42.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:42.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[64113]: Upgrade: Setting container_image for all osd 2024-09-20T22:46:42.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[64113]: Upgrade: Setting require_osd_release to 19 squid 2024-09-20T22:46:42.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:42.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[64113]: Upgrade: Setting container_image for all mds 2024-09-20T22:46:42.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:42.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[64113]: pgmap v207: 161 pgs: 26 peering, 24 active+undersized, 13 active+undersized+degraded, 98 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 47/630 objects degraded (7.460%) 2024-09-20T22:46:42.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:42.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[66621]: Upgrade: Setting container_image for all osd 2024-09-20T22:46:42.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[66621]: Upgrade: Setting require_osd_release to 19 squid 2024-09-20T22:46:42.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:42.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[66621]: Upgrade: Setting container_image for all mds 2024-09-20T22:46:42.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:42.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:42 smithi115 ceph-mon[66621]: pgmap v207: 161 pgs: 26 peering, 24 active+undersized, 13 active+undersized+degraded, 98 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 47/630 objects degraded (7.460%) 2024-09-20T22:46:42.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:42 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:42.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:42 smithi153 ceph-mon[56592]: Upgrade: Setting container_image for all osd 2024-09-20T22:46:42.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:42 smithi153 ceph-mon[56592]: Upgrade: Setting require_osd_release to 19 squid 2024-09-20T22:46:42.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:42 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:42.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:42 smithi153 ceph-mon[56592]: Upgrade: Setting container_image for all mds 2024-09-20T22:46:42.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:42 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:42.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:42 smithi153 ceph-mon[56592]: pgmap v207: 161 pgs: 26 peering, 24 active+undersized, 13 active+undersized+degraded, 98 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 47/630 objects degraded (7.460%) 2024-09-20T22:46:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:43 smithi153 ceph-mon[56592]: Upgrade: Updating rgw.foo.smithi115.cotebr (1/4) 2024-09-20T22:46:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:43 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:43 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:46:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:43 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:43 smithi153 ceph-mon[56592]: Deploying daemon rgw.foo.smithi115.cotebr on smithi115 2024-09-20T22:46:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:43 smithi153 ceph-mon[56592]: Health check update: Degraded data redundancy: 47/630 objects degraded (7.460%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T22:46:44.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:43 smithi153 ceph-mon[56592]: pgmap v208: 161 pgs: 26 peering, 17 active+undersized, 10 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40/630 objects degraded (6.349%) 2024-09-20T22:46:44.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[64113]: Upgrade: Updating rgw.foo.smithi115.cotebr (1/4) 2024-09-20T22:46:44.314 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:44.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:46:44.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:44.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[64113]: Deploying daemon rgw.foo.smithi115.cotebr on smithi115 2024-09-20T22:46:44.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[64113]: Health check update: Degraded data redundancy: 47/630 objects degraded (7.460%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T22:46:44.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[64113]: pgmap v208: 161 pgs: 26 peering, 17 active+undersized, 10 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40/630 objects degraded (6.349%) 2024-09-20T22:46:44.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[66621]: Upgrade: Updating rgw.foo.smithi115.cotebr (1/4) 2024-09-20T22:46:44.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:44.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi115.cotebr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:46:44.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:44.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[66621]: Deploying daemon rgw.foo.smithi115.cotebr on smithi115 2024-09-20T22:46:44.315 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[66621]: Health check update: Degraded data redundancy: 47/630 objects degraded (7.460%), 13 pgs degraded (PG_DEGRADED) 2024-09-20T22:46:44.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:43 smithi115 ceph-mon[66621]: pgmap v208: 161 pgs: 26 peering, 17 active+undersized, 10 active+undersized+degraded, 108 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40/630 objects degraded (6.349%) 2024-09-20T22:46:45.470 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:46:45 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:46:45] "GET /metrics HTTP/1.1" 200 38589 "" "Prometheus/2.43.0" 2024-09-20T22:46:46.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:45 smithi115 ceph-mon[66621]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/630 objects degraded (6.349%), 10 pgs degraded) 2024-09-20T22:46:46.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:45 smithi115 ceph-mon[66621]: Cluster is now healthy 2024-09-20T22:46:46.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:45 smithi115 ceph-mon[64113]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/630 objects degraded (6.349%), 10 pgs degraded) 2024-09-20T22:46:46.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:45 smithi115 ceph-mon[64113]: Cluster is now healthy 2024-09-20T22:46:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:45 smithi153 ceph-mon[56592]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/630 objects degraded (6.349%), 10 pgs degraded) 2024-09-20T22:46:46.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:45 smithi153 ceph-mon[56592]: Cluster is now healthy 2024-09-20T22:46:47.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:46 smithi115 ceph-mon[64113]: pgmap v209: 161 pgs: 19 peering, 142 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 746 B/s rd, 0 op/s 2024-09-20T22:46:47.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:46 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:47.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:46 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:47.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:46 smithi115 ceph-mon[66621]: pgmap v209: 161 pgs: 19 peering, 142 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 746 B/s rd, 0 op/s 2024-09-20T22:46:47.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:46 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:47.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:46 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:47.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:46 smithi153 ceph-mon[56592]: pgmap v209: 161 pgs: 19 peering, 142 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 746 B/s rd, 0 op/s 2024-09-20T22:46:47.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:46 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:47.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:46 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:49.081 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:48 smithi153 ceph-mon[56592]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T22:46:49.081 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:48 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:49.081 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:48 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:46:49.081 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:48 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:48 smithi115 ceph-mon[64113]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T22:46:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:48 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:48 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:46:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:48 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:49.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:48 smithi115 ceph-mon[66621]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T22:46:49.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:48 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:49.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:48 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi115.pccgis", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:46:49.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:48 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:50.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:49 smithi153 ceph-mon[56592]: Upgrade: Updating rgw.smpl.smithi115.pccgis (2/4) 2024-09-20T22:46:50.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:49 smithi153 ceph-mon[56592]: Deploying daemon rgw.smpl.smithi115.pccgis on smithi115 2024-09-20T22:46:50.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:50.098 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:50.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:49 smithi115 ceph-mon[64113]: Upgrade: Updating rgw.smpl.smithi115.pccgis (2/4) 2024-09-20T22:46:50.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:49 smithi115 ceph-mon[64113]: Deploying daemon rgw.smpl.smithi115.pccgis on smithi115 2024-09-20T22:46:50.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:50.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:49 smithi115 ceph-mon[66621]: Upgrade: Updating rgw.smpl.smithi115.pccgis (2/4) 2024-09-20T22:46:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:49 smithi115 ceph-mon[66621]: Deploying daemon rgw.smpl.smithi115.pccgis on smithi115 2024-09-20T22:46:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:50.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:50.496 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:46:51.125 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:50 smithi115 ceph-mon[64113]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 30 op/s 2024-09-20T22:46:51.125 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:50 smithi115 ceph-mon[66621]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 30 op/s 2024-09-20T22:46:51.176 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:46:51.177 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (6m) 2m ago 11m 26.0M - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:46:51.177 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (6m) 13s ago 11m 52.8M - dad864ee21e9 568386e3ebfd 2024-09-20T22:46:51.177 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (6m) 2m ago 10m 52.7M - 3.5 e1d6a67b021e e7840bcf555a 2024-09-20T22:46:51.177 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (5m) 13s ago 16m 441M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:46:51.177 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (6m) 2m ago 17m 507M - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:46:51.177 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (5m) 2m ago 18m 46.4M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:46:51.177 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (5m) 13s ago 16m 40.6M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 f4dde7cfd242 2024-09-20T22:46:51.177 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5m) 2m ago 16m 38.3M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 887b72a1971e 2024-09-20T22:46:51.177 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (7m) 2m ago 12m 14.7M - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:46:51.177 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (6m) 13s ago 12m 15.4M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:46:51.177 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4m) 2m ago 16m 45.3M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 98478f94c84e 2024-09-20T22:46:51.177 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (3m) 2m ago 15m 53.7M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 4c2f3b5efcd0 2024-09-20T22:46:51.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3m) 2m ago 15m 50.0M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 b90ea318f712 2024-09-20T22:46:51.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (2m) 2m ago 14m 11.1M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 9eec1f15261b 2024-09-20T22:46:51.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (2m) 13s ago 14m 57.4M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 ba7575091ec9 2024-09-20T22:46:51.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (87s) 13s ago 14m 46.5M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 b1f12957e36e 2024-09-20T22:46:51.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (52s) 13s ago 13m 53.0M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 c7f1c8d06b01 2024-09-20T22:46:51.178 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (18s) 13s ago 13m 11.9M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 ca686c937f42 2024-09-20T22:46:51.178 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (5m) 13s ago 12m 69.0M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:46:51.178 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 starting - - - - 2024-09-20T22:46:51.178 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (10m) 13s ago 10m 76.2M - 17.2.0 e1d6a67b021e f1cbf9052295 2024-09-20T22:46:51.178 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (10m) 2m ago 10m 74.1M - 17.2.0 e1d6a67b021e b077822c00b1 2024-09-20T22:46:51.178 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (10m) 13s ago 10m 72.6M - 17.2.0 e1d6a67b021e 2004e31daacc 2024-09-20T22:46:51.264 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:50 smithi153 ceph-mon[56592]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 30 op/s 2024-09-20T22:46:51.614 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:46:51.615 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:46:51.615 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:46:51.615 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:46:51.615 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:46:51.615 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:46:51.615 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:46:51.615 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:46:51.615 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 8 2024-09-20T22:46:51.615 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:46:51.615 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:46:51.615 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-09-20T22:46:51.615 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T22:46:51.615 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:46:51.616 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:46:51.616 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-09-20T22:46:51.616 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 14 2024-09-20T22:46:51.616 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:46:51.616 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:46:52.011 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:46:52.011 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:46:52.011 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:46:52.011 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:46:52.011 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:46:52.011 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr", 2024-09-20T22:46:52.011 INFO:teuthology.orchestra.run.smithi115.stdout: "osd", 2024-09-20T22:46:52.011 INFO:teuthology.orchestra.run.smithi115.stdout: "mon" 2024-09-20T22:46:52.011 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:46:52.011 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "13/23 daemons upgraded", 2024-09-20T22:46:52.011 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading rgw daemons", 2024-09-20T22:46:52.011 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:46:52.012 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:46:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:51 smithi115 ceph-mon[64113]: from='client.54496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:51 smithi115 ceph-mon[64113]: from='client.54502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:52.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:51 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3362833896' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:52.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:51 smithi115 ceph-mon[66621]: from='client.54496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:52.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:51 smithi115 ceph-mon[66621]: from='client.54502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:52.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:51 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/3362833896' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:52.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:51 smithi153 ceph-mon[56592]: from='client.54496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:52.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:51 smithi153 ceph-mon[56592]: from='client.54502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:52.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:51 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/3362833896' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:46:52.449 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:46:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:52 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:52 smithi153 ceph-mon[56592]: from='client.54508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:52 smithi153 ceph-mon[56592]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 98 B/s wr, 85 op/s 2024-09-20T22:46:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:52 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:52 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:53.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:52 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/1441859014' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:46:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:52 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:52 smithi115 ceph-mon[64113]: from='client.54508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:52 smithi115 ceph-mon[64113]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 98 B/s wr, 85 op/s 2024-09-20T22:46:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:52 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:52 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:53.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:52 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/1441859014' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:46:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:52 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:52 smithi115 ceph-mon[66621]: from='client.54508 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:52 smithi115 ceph-mon[66621]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 98 B/s wr, 85 op/s 2024-09-20T22:46:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:52 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:52 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:53.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:52 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/1441859014' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:46:54.128 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:53 smithi153 ceph-mon[56592]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:54.128 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:53 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:54.128 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:53 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:46:54.128 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:53 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:53 smithi115 ceph-mon[64113]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:53 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:53 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:46:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:53 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:54.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:53 smithi115 ceph-mon[66621]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:46:54.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:53 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:54.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:53 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.vetrnf", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:46:54.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:53 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:54.967 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:54 smithi153 ceph-mon[56592]: Upgrade: Updating rgw.foo.smithi153.vetrnf (3/4) 2024-09-20T22:46:54.968 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:54 smithi153 ceph-mon[56592]: Deploying daemon rgw.foo.smithi153.vetrnf on smithi153 2024-09-20T22:46:54.968 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:54 smithi153 ceph-mon[56592]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T22:46:54.968 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:54 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:54 smithi115 ceph-mon[64113]: Upgrade: Updating rgw.foo.smithi153.vetrnf (3/4) 2024-09-20T22:46:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:54 smithi115 ceph-mon[64113]: Deploying daemon rgw.foo.smithi153.vetrnf on smithi153 2024-09-20T22:46:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:54 smithi115 ceph-mon[64113]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T22:46:55.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:54 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:55.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:54 smithi115 ceph-mon[66621]: Upgrade: Updating rgw.foo.smithi153.vetrnf (3/4) 2024-09-20T22:46:55.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:54 smithi115 ceph-mon[66621]: Deploying daemon rgw.foo.smithi153.vetrnf on smithi153 2024-09-20T22:46:55.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:54 smithi115 ceph-mon[66621]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T22:46:55.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:54 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:55.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:46:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:46:55] "GET /metrics HTTP/1.1" 200 38588 "" "Prometheus/2.43.0" 2024-09-20T22:46:57.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:56 smithi115 ceph-mon[64113]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 85 B/s wr, 118 op/s 2024-09-20T22:46:57.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:56 smithi115 ceph-mon[66621]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 85 B/s wr, 118 op/s 2024-09-20T22:46:57.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:56 smithi153 ceph-mon[56592]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 85 B/s wr, 118 op/s 2024-09-20T22:46:59.019 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:58 smithi153 ceph-mon[56592]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T22:46:59.019 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:58 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:59.019 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:46:58 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:59.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:58 smithi115 ceph-mon[64113]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T22:46:59.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:58 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:59.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:46:58 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:59.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:58 smithi115 ceph-mon[66621]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T22:46:59.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:58 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:46:59.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:46:58 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:00.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:00 smithi115 ceph-mon[64113]: Upgrade: Updating rgw.smpl.smithi153.oznbtc (4/4) 2024-09-20T22:47:00.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:00 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:00.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:00 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:47:00.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:00 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:00.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:00 smithi115 ceph-mon[64113]: Deploying daemon rgw.smpl.smithi153.oznbtc on smithi153 2024-09-20T22:47:00.655 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:00 smithi115 ceph-mon[64113]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 170 B/s wr, 186 op/s 2024-09-20T22:47:00.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:00 smithi115 ceph-mon[66621]: Upgrade: Updating rgw.smpl.smithi153.oznbtc (4/4) 2024-09-20T22:47:00.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:00 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:00.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:00 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:47:00.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:00 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:00.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:00 smithi115 ceph-mon[66621]: Deploying daemon rgw.smpl.smithi153.oznbtc on smithi153 2024-09-20T22:47:00.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:00 smithi115 ceph-mon[66621]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 170 B/s wr, 186 op/s 2024-09-20T22:47:00.658 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:00 smithi153 ceph-mon[56592]: Upgrade: Updating rgw.smpl.smithi153.oznbtc (4/4) 2024-09-20T22:47:00.658 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:00 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:00.658 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:00 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi153.oznbtc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:47:00.658 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:00 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:00.658 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:00 smithi153 ceph-mon[56592]: Deploying daemon rgw.smpl.smithi153.oznbtc on smithi153 2024-09-20T22:47:00.658 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:00 smithi153 ceph-mon[56592]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 170 B/s wr, 186 op/s 2024-09-20T22:47:02.958 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:02 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:02.959 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:02 smithi153 ceph-mon[56592]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 255 B/s wr, 216 op/s 2024-09-20T22:47:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:02 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:03.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:02 smithi115 ceph-mon[64113]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 255 B/s wr, 216 op/s 2024-09-20T22:47:03.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:02 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:03.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:02 smithi115 ceph-mon[66621]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 255 B/s wr, 216 op/s 2024-09-20T22:47:04.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:04.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:04.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:04 smithi115 ceph-mon[64113]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 170 B/s wr, 178 op/s 2024-09-20T22:47:04.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:04.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:04.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:04 smithi115 ceph-mon[66621]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 170 B/s wr, 178 op/s 2024-09-20T22:47:04.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:04.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:04.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:04 smithi153 ceph-mon[56592]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 170 B/s wr, 178 op/s 2024-09-20T22:47:05.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:05 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:05.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:05 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:05.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:47:05 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:47:05] "GET /metrics HTTP/1.1" 200 38586 "" "Prometheus/2.43.0" 2024-09-20T22:47:05.426 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:05 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:06.428 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:06 smithi115 ceph-mon[64113]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 170 B/s wr, 202 op/s 2024-09-20T22:47:06.429 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:06 smithi115 ceph-mon[66621]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 170 B/s wr, 202 op/s 2024-09-20T22:47:06.528 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:06 smithi153 ceph-mon[56592]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 170 B/s wr, 202 op/s 2024-09-20T22:47:07.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:07 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:07 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:07 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:07 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:07 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:07 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:07 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:07 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:07 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:07 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:07 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:07 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:07 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:07 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:07 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:07 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:07 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:07.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:07 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:08.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:08 smithi115 ceph-mon[64113]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 255 B/s wr, 218 op/s 2024-09-20T22:47:08.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:08 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:08.654 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:08 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:08.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:08 smithi115 ceph-mon[66621]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 255 B/s wr, 218 op/s 2024-09-20T22:47:08.654 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:08 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:08.655 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:08 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:08.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:08 smithi153 ceph-mon[56592]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 255 B/s wr, 218 op/s 2024-09-20T22:47:08.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:08 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:08.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:08 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:10.954 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:10 smithi115 ceph-mon[64113]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T22:47:10.954 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:10 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:10.954 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:10 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:10.954 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:10 smithi115 ceph-mon[66621]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T22:47:10.954 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:10 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:10.954 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:10 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:10 smithi153 ceph-mon[56592]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-09-20T22:47:11.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:10 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:10 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: Detected new or changed devices on smithi153 2024-09-20T22:47:11.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi115.cotebr"}]: dispatch 2024-09-20T22:47:11.905 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi115.cotebr"}]': finished 2024-09-20T22:47:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi153.vetrnf"}]: dispatch 2024-09-20T22:47:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi153.vetrnf"}]': finished 2024-09-20T22:47:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi115.pccgis"}]: dispatch 2024-09-20T22:47:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi115.pccgis"}]': finished 2024-09-20T22:47:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi153.oznbtc"}]: dispatch 2024-09-20T22:47:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi153.oznbtc"}]': finished 2024-09-20T22:47:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.906 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: Detected new or changed devices on smithi153 2024-09-20T22:47:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:47:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:47:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.907 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi115.cotebr"}]: dispatch 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi115.cotebr"}]': finished 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi153.vetrnf"}]: dispatch 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi153.vetrnf"}]': finished 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi115.pccgis"}]: dispatch 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi115.pccgis"}]': finished 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi153.oznbtc"}]: dispatch 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi153.oznbtc"}]': finished 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.908 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:11.909 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:11 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: Detected new or changed devices on smithi153 2024-09-20T22:47:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:47:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:47:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:12.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:12.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:12.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:12.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:12.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi115.cotebr"}]: dispatch 2024-09-20T22:47:12.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi115.cotebr"}]': finished 2024-09-20T22:47:12.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi153.vetrnf"}]: dispatch 2024-09-20T22:47:12.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi153.vetrnf"}]': finished 2024-09-20T22:47:12.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi115.pccgis"}]: dispatch 2024-09-20T22:47:12.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi115.pccgis"}]': finished 2024-09-20T22:47:12.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi153.oznbtc"}]: dispatch 2024-09-20T22:47:12.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi153.oznbtc"}]': finished 2024-09-20T22:47:12.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:12.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:12.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:12.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:12.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:11 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:13.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:13.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[64113]: Detected new or changed devices on smithi115 2024-09-20T22:47:13.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[64113]: Upgrade: Setting container_image for all rgw 2024-09-20T22:47:13.118 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[64113]: Upgrade: Setting container_image for all rbd-mirror 2024-09-20T22:47:13.119 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[64113]: Upgrade: Setting container_image for all ceph-exporter 2024-09-20T22:47:13.119 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[64113]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 170 B/s wr, 147 op/s 2024-09-20T22:47:13.119 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:13.119 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:47:13.119 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:13.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:13.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[66621]: Detected new or changed devices on smithi115 2024-09-20T22:47:13.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[66621]: Upgrade: Setting container_image for all rgw 2024-09-20T22:47:13.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[66621]: Upgrade: Setting container_image for all rbd-mirror 2024-09-20T22:47:13.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[66621]: Upgrade: Setting container_image for all ceph-exporter 2024-09-20T22:47:13.119 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[66621]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 170 B/s wr, 147 op/s 2024-09-20T22:47:13.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:13.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:47:13.120 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:12 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:12 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:12 smithi153 ceph-mon[56592]: Detected new or changed devices on smithi115 2024-09-20T22:47:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:12 smithi153 ceph-mon[56592]: Upgrade: Setting container_image for all rgw 2024-09-20T22:47:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:12 smithi153 ceph-mon[56592]: Upgrade: Setting container_image for all rbd-mirror 2024-09-20T22:47:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:12 smithi153 ceph-mon[56592]: Upgrade: Setting container_image for all ceph-exporter 2024-09-20T22:47:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:12 smithi153 ceph-mon[56592]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 170 B/s wr, 147 op/s 2024-09-20T22:47:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:12 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:12 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi115.hwguxp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:47:13.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:12 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:13.930 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:13 smithi115 ceph-mon[64113]: Upgrade: Updating iscsi.foo.smithi115.hwguxp 2024-09-20T22:47:13.930 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:13 smithi115 ceph-mon[64113]: Deploying daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:47:13.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:13 smithi115 ceph-mon[66621]: Upgrade: Updating iscsi.foo.smithi115.hwguxp 2024-09-20T22:47:13.930 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:13 smithi115 ceph-mon[66621]: Deploying daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:47:14.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:13 smithi153 ceph-mon[56592]: Upgrade: Updating iscsi.foo.smithi115.hwguxp 2024-09-20T22:47:14.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:13 smithi153 ceph-mon[56592]: Deploying daemon iscsi.foo.smithi115.hwguxp on smithi115 2024-09-20T22:47:14.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:14 smithi115 ceph-mon[64113]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 85 B/s wr, 96 op/s 2024-09-20T22:47:14.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:14 smithi115 ceph-mon[66621]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 85 B/s wr, 96 op/s 2024-09-20T22:47:15.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:14 smithi153 ceph-mon[56592]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 85 B/s wr, 96 op/s 2024-09-20T22:47:15.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:47:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:47:15] "GET /metrics HTTP/1.1" 200 38586 "" "Prometheus/2.43.0" 2024-09-20T22:47:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:16 smithi115 ceph-mon[64113]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T22:47:17.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:16 smithi115 ceph-mon[66621]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T22:47:17.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:16 smithi153 ceph-mon[56592]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-09-20T22:47:19.081 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:18 smithi153 ceph-mon[56592]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 85 B/s wr, 50 op/s 2024-09-20T22:47:19.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:18 smithi115 ceph-mon[64113]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 85 B/s wr, 50 op/s 2024-09-20T22:47:19.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:18 smithi115 ceph-mon[66621]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 85 B/s wr, 50 op/s 2024-09-20T22:47:20.097 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:20.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:21.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:20 smithi115 ceph-mon[64113]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:21.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:20 smithi115 ceph-mon[66621]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:21.247 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:20 smithi153 ceph-mon[56592]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:22.817 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:47:23.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:22 smithi115 ceph-mon[64113]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:23.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:22 smithi115 ceph-mon[64113]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:23.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:22 smithi115 ceph-mon[66621]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:23.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:22 smithi115 ceph-mon[66621]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:23.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:22 smithi153 ceph-mon[56592]: from='client.25291 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:23.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:22 smithi153 ceph-mon[56592]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:23.471 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:47:23.471 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (7m) 16s ago 12m 26.1M - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:47:23.471 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (7m) 17s ago 12m 51.6M - dad864ee21e9 568386e3ebfd 2024-09-20T22:47:23.471 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 running (6m) 16s ago 11m 60.4M - 3.5 e1d6a67b021e e7840bcf555a 2024-09-20T22:47:23.471 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (6m) 17s ago 16m 442M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:47:23.471 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (6m) 16s ago 18m 511M - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:47:23.471 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6m) 16s ago 18m 54.4M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:47:23.471 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (5m) 17s ago 17m 42.9M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 f4dde7cfd242 2024-09-20T22:47:23.471 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (5m) 16s ago 17m 45.7M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 887b72a1971e 2024-09-20T22:47:23.471 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (7m) 16s ago 12m 14.9M - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:47:23.471 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (7m) 17s ago 12m 15.7M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:47:23.471 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (4m) 16s ago 16m 56.7M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 98478f94c84e 2024-09-20T22:47:23.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4m) 16s ago 16m 65.0M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 4c2f3b5efcd0 2024-09-20T22:47:23.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (3m) 16s ago 15m 59.1M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 b90ea318f712 2024-09-20T22:47:23.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3m) 16s ago 15m 54.5M 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 9eec1f15261b 2024-09-20T22:47:23.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (2m) 17s ago 14m 62.7M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 ba7575091ec9 2024-09-20T22:47:23.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (119s) 17s ago 14m 51.1M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 b1f12957e36e 2024-09-20T22:47:23.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (85s) 17s ago 14m 57.3M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 c7f1c8d06b01 2024-09-20T22:47:23.472 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (50s) 17s ago 13m 57.4M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 ca686c937f42 2024-09-20T22:47:23.472 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (6m) 17s ago 13m 69.1M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:47:23.472 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (37s) 16s ago 11m 75.9M - 19.1.1-367-g9e2ac70b cc9251b744a3 24a10397fee6 2024-09-20T22:47:23.472 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (26s) 17s ago 11m 75.0M - 19.1.1-367-g9e2ac70b cc9251b744a3 f150137b0d72 2024-09-20T22:47:23.472 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (31s) 16s ago 11m 78.1M - 19.1.1-367-g9e2ac70b cc9251b744a3 e32a7fce1447 2024-09-20T22:47:23.472 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (21s) 17s ago 11m 77.9M - 19.1.1-367-g9e2ac70b cc9251b744a3 93d547a5aa60 2024-09-20T22:47:23.879 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:47:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:47:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:47:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:47:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:47:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:47:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:47:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:47:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 8 2024-09-20T22:47:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:47:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:47:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 4 2024-09-20T22:47:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:47:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:47:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 17 2024-09-20T22:47:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:47:23.880 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:47:24.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:23 smithi115 ceph-mon[64113]: from='client.54601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:24.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:23 smithi115 ceph-mon[66621]: from='client.54601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:24.212 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:47:24.212 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:47:24.212 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:47:24.213 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:47:24.213 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:47:24.213 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr", 2024-09-20T22:47:24.213 INFO:teuthology.orchestra.run.smithi115.stdout: "osd", 2024-09-20T22:47:24.213 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw", 2024-09-20T22:47:24.213 INFO:teuthology.orchestra.run.smithi115.stdout: "mon" 2024-09-20T22:47:24.213 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:47:24.213 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:47:24.213 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Currently upgrading iscsi daemons", 2024-09-20T22:47:24.213 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": false 2024-09-20T22:47:24.213 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:47:24.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:23 smithi153 ceph-mon[56592]: from='client.54601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:24.624 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_OK 2024-09-20T22:47:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:24 smithi115 ceph-mon[64113]: from='client.44670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:25.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:24 smithi115 ceph-mon[64113]: from='client.44676 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:25.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:24 smithi115 ceph-mon[64113]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:25.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:24 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3804352760' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:25.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:24 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/2418350241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:47:25.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:24 smithi115 ceph-mon[66621]: from='client.44670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:25.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:24 smithi115 ceph-mon[66621]: from='client.44676 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:25.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:24 smithi115 ceph-mon[66621]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:25.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:24 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/3804352760' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:25.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:24 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/2418350241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:47:25.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:24 smithi153 ceph-mon[56592]: from='client.44670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:24 smithi153 ceph-mon[56592]: from='client.44676 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:24 smithi153 ceph-mon[56592]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:24 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/3804352760' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:47:25.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:24 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/2418350241' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:47:25.460 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:47:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:47:25] "GET /metrics HTTP/1.1" 200 38651 "" "Prometheus/2.43.0" 2024-09-20T22:47:26.040 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:25 smithi115 ceph-mon[64113]: from='client.54625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:26.040 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:25 smithi115 ceph-mon[66621]: from='client.54625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:26.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:25 smithi153 ceph-mon[56592]: from='client.54625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:26.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:47:26 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:47:26.073+0000 7f257adde640 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi115.hwguxp on host smithi115 failed. 2024-09-20T22:47:27.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:26 smithi153 ceph-mon[56592]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:27.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:26 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:27.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:26 smithi115 ceph-mon[64113]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:27.030 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:26 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:27.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:26 smithi115 ceph-mon[66621]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:47:27.030 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:26 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:28.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:27 smithi153 ceph-mon[56592]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi115.hwguxp on host smithi115 failed. 2024-09-20T22:47:28.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:27 smithi153 ceph-mon[56592]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:47:28.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:27 smithi153 ceph-mon[56592]: Health check failed: Upgrading daemon iscsi.foo.smithi115.hwguxp on host smithi115 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-09-20T22:47:28.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:27 smithi115 ceph-mon[64113]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi115.hwguxp on host smithi115 failed. 2024-09-20T22:47:28.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:27 smithi115 ceph-mon[64113]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:47:28.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:27 smithi115 ceph-mon[64113]: Health check failed: Upgrading daemon iscsi.foo.smithi115.hwguxp on host smithi115 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-09-20T22:47:28.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:27 smithi115 ceph-mon[66621]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi115.hwguxp on host smithi115 failed. 2024-09-20T22:47:28.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:27 smithi115 ceph-mon[66621]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:47:28.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:27 smithi115 ceph-mon[66621]: Health check failed: Upgrading daemon iscsi.foo.smithi115.hwguxp on host smithi115 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-09-20T22:47:30.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:29 smithi115 ceph-mon[66621]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 890 B/s rd, 0 op/s 2024-09-20T22:47:30.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:29 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:30.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:29 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:30.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:29 smithi115 ceph-mon[64113]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 890 B/s rd, 0 op/s 2024-09-20T22:47:30.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:29 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:30.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:29 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:30.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:29 smithi153 ceph-mon[56592]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 890 B/s rd, 0 op/s 2024-09-20T22:47:30.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:29 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:30.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:29 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:47:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:31 smithi153 ceph-mon[56592]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 894 B/s rd, 0 op/s 2024-09-20T22:47:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:31 smithi153 ceph-mon[56592]: Checking dashboard <-> RGW credentials 2024-09-20T22:47:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:47:31.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:31 smithi153 ceph-mon[56592]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T22:47:31.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:31 smithi153 ceph-mon[56592]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:47:31.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:31.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:31.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:31.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:31.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:47:31.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[64113]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 894 B/s rd, 0 op/s 2024-09-20T22:47:31.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[64113]: Checking dashboard <-> RGW credentials 2024-09-20T22:47:31.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:31.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:31.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:47:31.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[64113]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T22:47:31.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[64113]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:47:31.405 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:31.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:31.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:31.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:31.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:47:31.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[66621]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 894 B/s rd, 0 op/s 2024-09-20T22:47:31.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[66621]: Checking dashboard <-> RGW credentials 2024-09-20T22:47:31.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:31.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:31.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:47:31.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[66621]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-20T22:47:31.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[66621]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:47:31.406 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:32.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:32 smithi153 ceph-mon[56592]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-20T22:47:32.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:32 smithi115 ceph-mon[64113]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-20T22:47:32.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:32 smithi115 ceph-mon[66621]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-20T22:47:33.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:33 smithi153 ceph-mon[56592]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 3 op/s 2024-09-20T22:47:33.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:33 smithi115 ceph-mon[64113]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 3 op/s 2024-09-20T22:47:33.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:33 smithi115 ceph-mon[66621]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 3 op/s 2024-09-20T22:47:35.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:47:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:47:35] "GET /metrics HTTP/1.1" 200 38806 "" "Prometheus/2.43.0" 2024-09-20T22:47:36.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:35 smithi153 ceph-mon[56592]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-20T22:47:36.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:35 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:36.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:35 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:35 smithi115 ceph-mon[64113]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-20T22:47:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:35 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:36.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:35 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:36.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:35 smithi115 ceph-mon[66621]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-20T22:47:36.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:35 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:47:36.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:35 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:38.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:37 smithi153 ceph-mon[56592]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-20T22:47:38.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:37 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/2475715252' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:47:38.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:37 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/1325055428' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2748452063"}]: dispatch 2024-09-20T22:47:38.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:37 smithi153 ceph-mon[56592]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2748452063"}]: dispatch 2024-09-20T22:47:38.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:37 smithi115 ceph-mon[64113]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-20T22:47:38.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:37 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/2475715252' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:47:38.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:37 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/1325055428' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2748452063"}]: dispatch 2024-09-20T22:47:38.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:37 smithi115 ceph-mon[64113]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2748452063"}]: dispatch 2024-09-20T22:47:38.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:37 smithi115 ceph-mon[66621]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-09-20T22:47:38.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:37 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/2475715252' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:47:38.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:37 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/1325055428' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2748452063"}]: dispatch 2024-09-20T22:47:38.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:37 smithi115 ceph-mon[66621]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2748452063"}]: dispatch 2024-09-20T22:47:39.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:38 smithi153 ceph-mon[56592]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2748452063"}]': finished 2024-09-20T22:47:39.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:38 smithi153 ceph-mon[56592]: osdmap e150: 8 total, 8 up, 8 in 2024-09-20T22:47:39.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:38 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/3560299566' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3807666859"}]: dispatch 2024-09-20T22:47:39.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:38 smithi153 ceph-mon[56592]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3807666859"}]: dispatch 2024-09-20T22:47:39.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:38 smithi115 ceph-mon[64113]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2748452063"}]': finished 2024-09-20T22:47:39.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:38 smithi115 ceph-mon[64113]: osdmap e150: 8 total, 8 up, 8 in 2024-09-20T22:47:39.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:38 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3560299566' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3807666859"}]: dispatch 2024-09-20T22:47:39.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:38 smithi115 ceph-mon[64113]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3807666859"}]: dispatch 2024-09-20T22:47:39.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:38 smithi115 ceph-mon[66621]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/2748452063"}]': finished 2024-09-20T22:47:39.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:38 smithi115 ceph-mon[66621]: osdmap e150: 8 total, 8 up, 8 in 2024-09-20T22:47:39.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:38 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/3560299566' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3807666859"}]: dispatch 2024-09-20T22:47:39.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:38 smithi115 ceph-mon[66621]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3807666859"}]: dispatch 2024-09-20T22:47:39.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:39 smithi115 ceph-mon[64113]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-09-20T22:47:39.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:39 smithi115 ceph-mon[64113]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3807666859"}]': finished 2024-09-20T22:47:39.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:39 smithi115 ceph-mon[64113]: osdmap e151: 8 total, 8 up, 8 in 2024-09-20T22:47:39.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:39 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/2159235573' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/585859721"}]: dispatch 2024-09-20T22:47:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:39 smithi115 ceph-mon[66621]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-09-20T22:47:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:39 smithi115 ceph-mon[66621]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3807666859"}]': finished 2024-09-20T22:47:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:39 smithi115 ceph-mon[66621]: osdmap e151: 8 total, 8 up, 8 in 2024-09-20T22:47:39.905 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:39 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/2159235573' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/585859721"}]: dispatch 2024-09-20T22:47:40.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:39 smithi153 ceph-mon[56592]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-09-20T22:47:40.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:39 smithi153 ceph-mon[56592]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3807666859"}]': finished 2024-09-20T22:47:40.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:39 smithi153 ceph-mon[56592]: osdmap e151: 8 total, 8 up, 8 in 2024-09-20T22:47:40.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:39 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/2159235573' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/585859721"}]: dispatch 2024-09-20T22:47:41.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:40 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/2159235573' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/585859721"}]': finished 2024-09-20T22:47:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:40 smithi153 ceph-mon[56592]: osdmap e152: 8 total, 8 up, 8 in 2024-09-20T22:47:41.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:40 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/198188583' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3845200438"}]: dispatch 2024-09-20T22:47:41.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:40 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/2159235573' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/585859721"}]': finished 2024-09-20T22:47:41.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:40 smithi115 ceph-mon[64113]: osdmap e152: 8 total, 8 up, 8 in 2024-09-20T22:47:41.045 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:40 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/198188583' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3845200438"}]: dispatch 2024-09-20T22:47:41.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:40 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/2159235573' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6801/585859721"}]': finished 2024-09-20T22:47:41.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:40 smithi115 ceph-mon[66621]: osdmap e152: 8 total, 8 up, 8 in 2024-09-20T22:47:41.045 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:40 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/198188583' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3845200438"}]: dispatch 2024-09-20T22:47:41.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:41 smithi115 ceph-mon[64113]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:47:41.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:41 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/198188583' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3845200438"}]': finished 2024-09-20T22:47:41.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:41 smithi115 ceph-mon[64113]: osdmap e153: 8 total, 8 up, 8 in 2024-09-20T22:47:41.991 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:41 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3761626130' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/585859721"}]: dispatch 2024-09-20T22:47:41.992 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:41 smithi115 ceph-mon[64113]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/585859721"}]: dispatch 2024-09-20T22:47:41.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:41 smithi115 ceph-mon[66621]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:47:41.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:41 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/198188583' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3845200438"}]': finished 2024-09-20T22:47:41.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:41 smithi115 ceph-mon[66621]: osdmap e153: 8 total, 8 up, 8 in 2024-09-20T22:47:41.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:41 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/3761626130' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/585859721"}]: dispatch 2024-09-20T22:47:41.992 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:41 smithi115 ceph-mon[66621]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/585859721"}]: dispatch 2024-09-20T22:47:42.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:41 smithi153 ceph-mon[56592]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:47:42.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:41 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/198188583' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/3845200438"}]': finished 2024-09-20T22:47:42.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:41 smithi153 ceph-mon[56592]: osdmap e153: 8 total, 8 up, 8 in 2024-09-20T22:47:42.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:41 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/3761626130' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/585859721"}]: dispatch 2024-09-20T22:47:42.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:41 smithi153 ceph-mon[56592]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/585859721"}]: dispatch 2024-09-20T22:47:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:42 smithi153 ceph-mon[56592]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/585859721"}]': finished 2024-09-20T22:47:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:42 smithi153 ceph-mon[56592]: osdmap e154: 8 total, 8 up, 8 in 2024-09-20T22:47:43.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:42 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/3290261538' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1495517185"}]: dispatch 2024-09-20T22:47:43.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:42 smithi115 ceph-mon[66621]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/585859721"}]': finished 2024-09-20T22:47:43.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:42 smithi115 ceph-mon[66621]: osdmap e154: 8 total, 8 up, 8 in 2024-09-20T22:47:43.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:42 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/3290261538' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1495517185"}]: dispatch 2024-09-20T22:47:43.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:42 smithi115 ceph-mon[64113]: from='client.? ' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:6800/585859721"}]': finished 2024-09-20T22:47:43.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:42 smithi115 ceph-mon[64113]: osdmap e154: 8 total, 8 up, 8 in 2024-09-20T22:47:43.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:42 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3290261538' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1495517185"}]: dispatch 2024-09-20T22:47:44.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:43 smithi153 ceph-mon[56592]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-20T22:47:44.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:43 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/3290261538' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1495517185"}]': finished 2024-09-20T22:47:44.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:43 smithi153 ceph-mon[56592]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T22:47:44.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:43 smithi115 ceph-mon[64113]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-20T22:47:44.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:43 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3290261538' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1495517185"}]': finished 2024-09-20T22:47:44.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:43 smithi115 ceph-mon[64113]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T22:47:44.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:43 smithi115 ceph-mon[66621]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-20T22:47:44.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:43 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/3290261538' entity='client.iscsi.foo.smithi115.hwguxp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.115:0/1495517185"}]': finished 2024-09-20T22:47:44.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:43 smithi115 ceph-mon[66621]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T22:47:45.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:47:45 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:47:45] "GET /metrics HTTP/1.1" 200 38806 "" "Prometheus/2.43.0" 2024-09-20T22:47:46.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:45 smithi153 ceph-mon[56592]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 1 op/s 2024-09-20T22:47:46.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:45 smithi115 ceph-mon[64113]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 1 op/s 2024-09-20T22:47:46.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:45 smithi115 ceph-mon[66621]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 1 op/s 2024-09-20T22:47:48.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:47 smithi153 ceph-mon[56592]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:47:48.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:47 smithi115 ceph-mon[64113]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:47:48.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:47 smithi115 ceph-mon[66621]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:47:49.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:48 smithi153 ceph-mon[56592]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:49.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:48 smithi115 ceph-mon[64113]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:49.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:48 smithi115 ceph-mon[66621]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:50.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:49 smithi153 ceph-mon[56592]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:50.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:50.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:49 smithi115 ceph-mon[64113]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:50.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:50.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:49 smithi115 ceph-mon[66621]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:50.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:52.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:51 smithi153 ceph-mon[56592]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-09-20T22:47:52.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:51 smithi115 ceph-mon[64113]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-09-20T22:47:52.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:51 smithi115 ceph-mon[66621]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-09-20T22:47:54.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:53 smithi153 ceph-mon[56592]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:47:54.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:53 smithi115 ceph-mon[64113]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:47:54.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:53 smithi115 ceph-mon[66621]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:47:54.988 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:47:55.323 INFO:teuthology.orchestra.run.smithi115.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi115.hwguxp on host smithi115 failed." 2024-09-20T22:47:55.323 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:47:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:47:55] "GET /metrics HTTP/1.1" 200 38805 "" "Prometheus/2.43.0" 2024-09-20T22:47:56.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:55 smithi153 ceph-mon[56592]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-09-20T22:47:56.060 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:55 smithi115 ceph-mon[64113]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-09-20T22:47:56.060 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:55 smithi115 ceph-mon[66621]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-09-20T22:47:56.084 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph orch ps' 2024-09-20T22:47:56.961 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:56 smithi115 ceph-mon[64113]: from='client.54733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:56.961 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:56 smithi115 ceph-mon[64113]: from='client.44802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:56.961 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:56 smithi115 ceph-mon[66621]: from='client.54733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:56.961 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:56 smithi115 ceph-mon[66621]: from='client.44802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:57.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:56 smithi153 ceph-mon[56592]: from='client.54733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:57.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:56 smithi153 ceph-mon[56592]: from='client.44802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:47:57.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:57 smithi115 ceph-mon[64113]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:57.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:57 smithi115 ceph-mon[66621]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:58.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:57 smithi153 ceph-mon[56592]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:47:58.039 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:47:58.039 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (8m) 28s ago 12m - - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:47:58.039 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (7m) 28s ago 12m 51.8M - dad864ee21e9 568386e3ebfd 2024-09-20T22:47:58.039 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 unknown 28s ago 11m - - 2024-09-20T22:47:58.039 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (6m) 28s ago 17m 442M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (7m) 28s ago 19m - - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (6m) 28s ago 19m - 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (6m) 28s ago 17m 41.5M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 f4dde7cfd242 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (6m) 28s ago 17m - 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 887b72a1971e 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (8m) 28s ago 13m - - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (7m) 28s ago 13m 15.2M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (5m) 28s ago 17m - 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 98478f94c84e 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (4m) 28s ago 16m - 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 4c2f3b5efcd0 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (4m) 28s ago 16m - 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 b90ea318f712 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (3m) 28s ago 15m - 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 9eec1f15261b 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (3m) 28s ago 15m 61.9M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 ba7575091ec9 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (2m) 28s ago 15m 50.3M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 b1f12957e36e 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (119s) 28s ago 14m 57.5M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 c7f1c8d06b01 2024-09-20T22:47:58.040 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (85s) 28s ago 14m 57.7M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 ca686c937f42 2024-09-20T22:47:58.041 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (7m) 28s ago 13m 69.1M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:47:58.041 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (71s) 28s ago 12m - - 19.1.1-367-g9e2ac70b cc9251b744a3 24a10397fee6 2024-09-20T22:47:58.041 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (60s) 28s ago 12m 75.2M - 19.1.1-367-g9e2ac70b cc9251b744a3 f150137b0d72 2024-09-20T22:47:58.041 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (66s) 28s ago 11m - - 19.1.1-367-g9e2ac70b cc9251b744a3 e32a7fce1447 2024-09-20T22:47:58.041 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (55s) 28s ago 11m 78.2M - 19.1.1-367-g9e2ac70b cc9251b744a3 93d547a5aa60 2024-09-20T22:47:58.741 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph versions' 2024-09-20T22:47:58.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:58 smithi115 ceph-mon[64113]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:58.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:58 smithi115 ceph-mon[66621]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:59.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:58 smithi153 ceph-mon[56592]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:00.008 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:59 smithi115 ceph-mon[64113]: from='client.54745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:00.008 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:47:59 smithi115 ceph-mon[64113]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:48:00.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:59 smithi115 ceph-mon[66621]: from='client.54745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:00.008 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:47:59 smithi115 ceph-mon[66621]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:48:00.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:59 smithi153 ceph-mon[56592]: from='client.54745 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:48:00.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:47:59 smithi153 ceph-mon[56592]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:48:00.809 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:48:00.809 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:48:00.809 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:48:00.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:48:00.809 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:48:00.809 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:48:00.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:48:00.810 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:48:00.810 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 8 2024-09-20T22:48:00.810 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:48:00.810 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:48:00.810 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 4 2024-09-20T22:48:00.810 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:48:00.810 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:48:00.810 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 17 2024-09-20T22:48:00.810 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:48:00.810 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:48:01.479 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-09-20T22:48:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:01 smithi153 ceph-mon[56592]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:02.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:01 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/2102799767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:48:02.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:01 smithi115 ceph-mon[66621]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:02.102 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:01 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/2102799767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:48:02.102 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:01 smithi115 ceph-mon[64113]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:02.103 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:01 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/2102799767' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:48:03.242 INFO:teuthology.orchestra.run.smithi115.stdout:wait for servicemap items w/ changing names to refresh 2024-09-20T22:48:03.863 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:03 smithi115 ceph-mon[64113]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:03.863 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:03 smithi115 ceph-mon[66621]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:03.887 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'sleep 60' 2024-09-20T22:48:04.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:03 smithi153 ceph-mon[56592]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:05.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:05.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:05.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:05.460 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:48:05 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:48:05] "GET /metrics HTTP/1.1" 200 38805 "" "Prometheus/2.43.0" 2024-09-20T22:48:06.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:05 smithi115 ceph-mon[64113]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:06.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:05 smithi115 ceph-mon[66621]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:06.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:05 smithi153 ceph-mon[56592]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:08.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:07 smithi115 ceph-mon[64113]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:08.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:07 smithi115 ceph-mon[66621]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:08.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:07 smithi153 ceph-mon[56592]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:09.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:08 smithi115 ceph-mon[64113]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:09.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:08 smithi115 ceph-mon[66621]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:09.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:08 smithi153 ceph-mon[56592]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:10.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:09 smithi115 ceph-mon[64113]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:10.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:09 smithi115 ceph-mon[66621]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:10.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:09 smithi153 ceph-mon[56592]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:11 smithi115 ceph-mon[64113]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:12.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:11 smithi115 ceph-mon[66621]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:12.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:11 smithi153 ceph-mon[56592]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:14.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:13 smithi115 ceph-mon[64113]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:14.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:13 smithi115 ceph-mon[66621]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:14.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:13 smithi153 ceph-mon[56592]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:15.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:48:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:48:15] "GET /metrics HTTP/1.1" 200 38805 "" "Prometheus/2.43.0" 2024-09-20T22:48:16.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:15 smithi115 ceph-mon[64113]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:16.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:15 smithi115 ceph-mon[66621]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:16.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:15 smithi153 ceph-mon[56592]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:18.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:17 smithi115 ceph-mon[64113]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:18.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:17 smithi115 ceph-mon[66621]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:18.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:17 smithi153 ceph-mon[56592]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:19.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:18 smithi115 ceph-mon[64113]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:19.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:18 smithi115 ceph-mon[66621]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:19.180 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:18 smithi153 ceph-mon[56592]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:20.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:19 smithi153 ceph-mon[56592]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:20.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:19 smithi115 ceph-mon[64113]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:20.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:20.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:19 smithi115 ceph-mon[66621]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:20.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:22.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:21 smithi115 ceph-mon[64113]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:22.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:21 smithi115 ceph-mon[66621]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:22.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:21 smithi153 ceph-mon[56592]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:24.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:23 smithi115 ceph-mon[64113]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:24.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:23 smithi115 ceph-mon[66621]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:24.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:23 smithi153 ceph-mon[56592]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:25.654 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:48:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:48:25] "GET /metrics HTTP/1.1" 200 38806 "" "Prometheus/2.43.0" 2024-09-20T22:48:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:25 smithi115 ceph-mon[64113]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:48:26.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:25 smithi115 ceph-mon[66621]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:48:26.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:25 smithi153 ceph-mon[56592]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:48:28.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:27 smithi115 ceph-mon[64113]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:28.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:27 smithi115 ceph-mon[66621]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:28.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:27 smithi153 ceph-mon[56592]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:29.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:28 smithi115 ceph-mon[64113]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:29.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:28 smithi115 ceph-mon[66621]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:29.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:28 smithi153 ceph-mon[56592]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:30.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:29 smithi115 ceph-mon[64113]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:48:30.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:29 smithi115 ceph-mon[66621]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:48:30.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:29 smithi153 ceph-mon[56592]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-20T22:48:32.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:31 smithi115 ceph-mon[64113]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:32.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:48:32.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:48:32.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:31 smithi115 ceph-mon[64113]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-09-20T22:48:32.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:31 smithi115 ceph-mon[64113]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-09-20T22:48:32.155 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:48:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:31 smithi115 ceph-mon[66621]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:48:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:48:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:31 smithi115 ceph-mon[66621]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-09-20T22:48:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:31 smithi115 ceph-mon[66621]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-09-20T22:48:32.155 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:48:32.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:31 smithi153 ceph-mon[56592]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:32.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:48:32.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:48:32.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:31 smithi153 ceph-mon[56592]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-09-20T22:48:32.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:31 smithi153 ceph-mon[56592]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-09-20T22:48:32.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:48:34.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:33 smithi153 ceph-mon[56592]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:34.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:33 smithi115 ceph-mon[64113]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:34.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:33 smithi115 ceph-mon[66621]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:35.202 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:34 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:35.202 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:34 smithi115 ceph-mon[64113]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-09-20T22:48:35.202 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:34 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:35.202 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:34 smithi115 ceph-mon[66621]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-09-20T22:48:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:34 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:34 smithi153 ceph-mon[56592]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-09-20T22:48:35.655 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:48:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:48:35] "GET /metrics HTTP/1.1" 200 38802 "" "Prometheus/2.43.0" 2024-09-20T22:48:37.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:36 smithi153 ceph-mon[56592]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:37.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:36 smithi115 ceph-mon[64113]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:37.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:36 smithi115 ceph-mon[66621]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:38.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:37 smithi153 ceph-mon[56592]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:38.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:37 smithi115 ceph-mon[64113]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:38.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:37 smithi115 ceph-mon[66621]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:39.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:38 smithi153 ceph-mon[56592]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:39.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:38 smithi115 ceph-mon[64113]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:39.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:38 smithi115 ceph-mon[66621]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:41.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:40 smithi153 ceph-mon[56592]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:48:41.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:40 smithi115 ceph-mon[64113]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:48:41.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:40 smithi115 ceph-mon[66621]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:48:43.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:42 smithi153 ceph-mon[56592]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:43.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:42 smithi115 ceph-mon[64113]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:43.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:42 smithi115 ceph-mon[66621]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:45.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:44 smithi153 ceph-mon[56592]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:45.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:44 smithi115 ceph-mon[64113]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:45.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:48:45 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:48:45] "GET /metrics HTTP/1.1" 200 38802 "" "Prometheus/2.43.0" 2024-09-20T22:48:45.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:44 smithi115 ceph-mon[66621]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:47.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:46 smithi153 ceph-mon[56592]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:47.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:46 smithi115 ceph-mon[64113]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:47.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:46 smithi115 ceph-mon[66621]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:48.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:47 smithi153 ceph-mon[56592]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:48.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:47 smithi115 ceph-mon[64113]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:48.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:47 smithi115 ceph-mon[66621]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:49.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:48 smithi153 ceph-mon[56592]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:49.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:48 smithi115 ceph-mon[64113]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:49.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:48 smithi115 ceph-mon[66621]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:50.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:49 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:50.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:49 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:50.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:49 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:51.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:50 smithi153 ceph-mon[56592]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:51.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:50 smithi115 ceph-mon[64113]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:51.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:50 smithi115 ceph-mon[66621]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:53.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:52 smithi153 ceph-mon[56592]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:53.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:52 smithi115 ceph-mon[64113]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:53.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:52 smithi115 ceph-mon[66621]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:55.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:54 smithi153 ceph-mon[56592]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:55.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:54 smithi115 ceph-mon[64113]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:55.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:48:55 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:48:55] "GET /metrics HTTP/1.1" 200 38802 "" "Prometheus/2.43.0" 2024-09-20T22:48:55.405 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:54 smithi115 ceph-mon[66621]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:57.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:56 smithi153 ceph-mon[56592]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:57.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:56 smithi115 ceph-mon[64113]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:57.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:56 smithi115 ceph-mon[66621]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:48:58.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:57 smithi153 ceph-mon[56592]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:58.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:57 smithi115 ceph-mon[64113]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:58.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:57 smithi115 ceph-mon[66621]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:59.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:48:58 smithi153 ceph-mon[56592]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:59.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:48:58 smithi115 ceph-mon[64113]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:48:59.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:48:58 smithi115 ceph-mon[66621]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:01.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:00 smithi153 ceph-mon[56592]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:01.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:00 smithi115 ceph-mon[64113]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:01.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:00 smithi115 ceph-mon[66621]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:03.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:02 smithi153 ceph-mon[56592]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:03.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:02 smithi115 ceph-mon[64113]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:03.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:02 smithi115 ceph-mon[66621]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:05.029 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:04 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:05.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:04 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:05.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:04 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:05.581 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:05 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:49:05] "GET /metrics HTTP/1.1" 200 38803 "" "Prometheus/2.43.0" 2024-09-20T22:49:05.903 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:05 smithi115 ceph-mon[64113]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:05.903 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:05 smithi115 ceph-mon[66621]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:06.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:05 smithi153 ceph-mon[56592]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:06.341 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph orch ps' 2024-09-20T22:49:07.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:06 smithi153 ceph-mon[56592]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:07.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:06 smithi115 ceph-mon[64113]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:07.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:06 smithi115 ceph-mon[66621]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:08.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:07 smithi115 ceph-mon[64113]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:08.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:07 smithi115 ceph-mon[66621]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:08.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:07 smithi153 ceph-mon[56592]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:08.361 INFO:teuthology.orchestra.run.smithi115.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T22:49:08.362 INFO:teuthology.orchestra.run.smithi115.stdout:alertmanager.a smithi115 *:9093,9094 running (9m) 98s ago 13m - - 0.25.0 c8568f914cd2 7b7065a4ae73 2024-09-20T22:49:08.362 INFO:teuthology.orchestra.run.smithi115.stdout:grafana.a smithi153 *:3000 running (8m) 99s ago 13m 51.8M - dad864ee21e9 568386e3ebfd 2024-09-20T22:49:08.362 INFO:teuthology.orchestra.run.smithi115.stdout:iscsi.foo.smithi115.hwguxp smithi115 unknown 98s ago 12m - - 2024-09-20T22:49:08.362 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.x smithi153 *:8443,9283,8765 running (8m) 99s ago 18m 442M - 19.1.1-367-g9e2ac70b cc9251b744a3 5de21dfaf787 2024-09-20T22:49:08.362 INFO:teuthology.orchestra.run.smithi115.stdout:mgr.y smithi115 *:8443,9283,8765 running (8m) 98s ago 20m - - 19.1.1-367-g9e2ac70b cc9251b744a3 07b943d2934a 2024-09-20T22:49:08.362 INFO:teuthology.orchestra.run.smithi115.stdout:mon.a smithi115 running (7m) 98s ago 20m - 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 29137eed6a0a 2024-09-20T22:49:08.362 INFO:teuthology.orchestra.run.smithi115.stdout:mon.b smithi153 running (7m) 99s ago 18m 41.5M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 f4dde7cfd242 2024-09-20T22:49:08.362 INFO:teuthology.orchestra.run.smithi115.stdout:mon.c smithi115 running (7m) 98s ago 19m - 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 887b72a1971e 2024-09-20T22:49:08.362 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.a smithi115 *:9100 running (9m) 98s ago 14m - - 1.5.0 0da6a335fe13 807c51f95c4d 2024-09-20T22:49:08.362 INFO:teuthology.orchestra.run.smithi115.stdout:node-exporter.b smithi153 *:9100 running (8m) 99s ago 14m 15.2M - 1.5.0 0da6a335fe13 3f057318ca36 2024-09-20T22:49:08.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.0 smithi115 running (6m) 98s ago 18m - 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 98478f94c84e 2024-09-20T22:49:08.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.1 smithi115 running (6m) 98s ago 17m - 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 4c2f3b5efcd0 2024-09-20T22:49:08.362 INFO:teuthology.orchestra.run.smithi115.stdout:osd.2 smithi115 running (5m) 98s ago 17m - 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 b90ea318f712 2024-09-20T22:49:08.363 INFO:teuthology.orchestra.run.smithi115.stdout:osd.3 smithi115 running (4m) 98s ago 17m - 2736M 19.1.1-367-g9e2ac70b cc9251b744a3 9eec1f15261b 2024-09-20T22:49:08.363 INFO:teuthology.orchestra.run.smithi115.stdout:osd.4 smithi153 running (4m) 99s ago 16m 61.9M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 ba7575091ec9 2024-09-20T22:49:08.363 INFO:teuthology.orchestra.run.smithi115.stdout:osd.5 smithi153 running (3m) 99s ago 16m 50.3M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 b1f12957e36e 2024-09-20T22:49:08.363 INFO:teuthology.orchestra.run.smithi115.stdout:osd.6 smithi153 running (3m) 99s ago 15m 57.5M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 c7f1c8d06b01 2024-09-20T22:49:08.363 INFO:teuthology.orchestra.run.smithi115.stdout:osd.7 smithi153 running (2m) 99s ago 15m 57.7M 2992M 19.1.1-367-g9e2ac70b cc9251b744a3 ca686c937f42 2024-09-20T22:49:08.363 INFO:teuthology.orchestra.run.smithi115.stdout:prometheus.a smithi153 *:9095 running (8m) 99s ago 14m 69.1M - 2.43.0 a07b618ecd1d cb068a98c110 2024-09-20T22:49:08.363 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi115.cotebr smithi115 *:8000 running (2m) 98s ago 13m - - 19.1.1-367-g9e2ac70b cc9251b744a3 24a10397fee6 2024-09-20T22:49:08.363 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.foo.smithi153.vetrnf smithi153 *:8000 running (2m) 99s ago 13m 75.2M - 19.1.1-367-g9e2ac70b cc9251b744a3 f150137b0d72 2024-09-20T22:49:08.363 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi115.pccgis smithi115 *:80 running (2m) 98s ago 13m - - 19.1.1-367-g9e2ac70b cc9251b744a3 e32a7fce1447 2024-09-20T22:49:08.363 INFO:teuthology.orchestra.run.smithi115.stdout:rgw.smpl.smithi153.oznbtc smithi153 *:80 running (2m) 99s ago 13m 78.2M - 19.1.1-367-g9e2ac70b cc9251b744a3 93d547a5aa60 2024-09-20T22:49:09.059 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph versions' 2024-09-20T22:49:09.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:08 smithi115 ceph-mon[64113]: from='client.44820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:09.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:08 smithi115 ceph-mon[64113]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:09.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:08 smithi115 ceph-mon[66621]: from='client.44820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:09.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:08 smithi115 ceph-mon[66621]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:09.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:08 smithi153 ceph-mon[56592]: from='client.44820 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:09.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:08 smithi153 ceph-mon[56592]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:11.192 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:49:11.192 INFO:teuthology.orchestra.run.smithi115.stdout: "mon": { 2024-09-20T22:49:11.192 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T22:49:11.192 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:49:11.192 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr": { 2024-09-20T22:49:11.192 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T22:49:11.192 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:49:11.192 INFO:teuthology.orchestra.run.smithi115.stdout: "osd": { 2024-09-20T22:49:11.192 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 8 2024-09-20T22:49:11.193 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:49:11.193 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw": { 2024-09-20T22:49:11.193 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 4 2024-09-20T22:49:11.193 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-09-20T22:49:11.193 INFO:teuthology.orchestra.run.smithi115.stdout: "overall": { 2024-09-20T22:49:11.193 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 17 2024-09-20T22:49:11.193 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-09-20T22:49:11.193 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:49:11.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:10 smithi153 ceph-mon[56592]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:11.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:10 smithi115 ceph-mon[64113]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:11.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:10 smithi115 ceph-mon[66621]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:11.924 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph orch upgrade status' 2024-09-20T22:49:12.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:11 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/1340858667' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:49:12.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:11 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/1340858667' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:49:12.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:11 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/1340858667' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:49:13.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:12 smithi153 ceph-mon[56592]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:13.315 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:12 smithi115 ceph-mon[64113]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:13.316 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:12 smithi115 ceph-mon[66621]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:14.017 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:49:14.017 INFO:teuthology.orchestra.run.smithi115.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T22:49:14.017 INFO:teuthology.orchestra.run.smithi115.stdout: "in_progress": true, 2024-09-20T22:49:14.017 INFO:teuthology.orchestra.run.smithi115.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T22:49:14.018 INFO:teuthology.orchestra.run.smithi115.stdout: "services_complete": [ 2024-09-20T22:49:14.018 INFO:teuthology.orchestra.run.smithi115.stdout: "mgr", 2024-09-20T22:49:14.018 INFO:teuthology.orchestra.run.smithi115.stdout: "osd", 2024-09-20T22:49:14.018 INFO:teuthology.orchestra.run.smithi115.stdout: "rgw", 2024-09-20T22:49:14.018 INFO:teuthology.orchestra.run.smithi115.stdout: "mon" 2024-09-20T22:49:14.018 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-09-20T22:49:14.018 INFO:teuthology.orchestra.run.smithi115.stdout: "progress": "17/23 daemons upgraded", 2024-09-20T22:49:14.018 INFO:teuthology.orchestra.run.smithi115.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi115.hwguxp on host smithi115 failed.", 2024-09-20T22:49:14.018 INFO:teuthology.orchestra.run.smithi115.stdout: "is_paused": true 2024-09-20T22:49:14.018 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:49:14.654 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph health detail' 2024-09-20T22:49:15.202 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:14 smithi115 ceph-mon[64113]: from='client.44832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:15.202 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:14 smithi115 ceph-mon[64113]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:15.202 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:14 smithi115 ceph-mon[66621]: from='client.44832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:15.202 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:14 smithi115 ceph-mon[66621]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:15.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:14 smithi153 ceph-mon[56592]: from='client.44832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:15.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:14 smithi153 ceph-mon[56592]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:15.637 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:15 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:49:15] "GET /metrics HTTP/1.1" 200 38803 "" "Prometheus/2.43.0" 2024-09-20T22:49:16.851 INFO:teuthology.orchestra.run.smithi115.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi115.hwguxp on host smithi115 failed. 2024-09-20T22:49:16.851 INFO:teuthology.orchestra.run.smithi115.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-20T22:49:16.851 INFO:teuthology.orchestra.run.smithi115.stdout: daemon iscsi.foo.smithi115.hwguxp on smithi115 is in unknown state 2024-09-20T22:49:16.851 INFO:teuthology.orchestra.run.smithi115.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi115.hwguxp on host smithi115 failed. 2024-09-20T22:49:16.851 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrade daemon: iscsi.foo.smithi115.hwguxp: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi115.hwguxp ... 2024-09-20T22:49:16.851 INFO:teuthology.orchestra.run.smithi115.stdout:Creating ceph-iscsi config... 2024-09-20T22:49:16.851 INFO:teuthology.orchestra.run.smithi115.stdout:Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/iscsi-gateway.cfg 2024-09-20T22:49:16.851 INFO:teuthology.orchestra.run.smithi115.stdout:Write file: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/iscsi.foo.smithi115.hwguxp/tcmu-runner-entrypoint.sh 2024-09-20T22:49:16.851 INFO:teuthology.orchestra.run.smithi115.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d9fe0f662\x2d779f\x2d11ef\x2dbaf5\x2defdc52797490.slice/ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service 2024-09-20T22:49:16.852 INFO:teuthology.orchestra.run.smithi115.stdout:Non-zero exit code 1 from systemctl start ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp 2024-09-20T22:49:16.852 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:49:16.852 INFO:teuthology.orchestra.run.smithi115.stdout:systemctl: stderr See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:49:16.852 INFO:teuthology.orchestra.run.smithi115.stdout:Traceback (most recent call last): 2024-09-20T22:49:16.852 INFO:teuthology.orchestra.run.smithi115.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-20T22:49:16.852 INFO:teuthology.orchestra.run.smithi115.stdout: return _run_code(code, main_globals, None, 2024-09-20T22:49:16.852 INFO:teuthology.orchestra.run.smithi115.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-20T22:49:16.852 INFO:teuthology.orchestra.run.smithi115.stdout: exec(code, run_globals) 2024-09-20T22:49:16.852 INFO:teuthology.orchestra.run.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5579, in 2024-09-20T22:49:16.852 INFO:teuthology.orchestra.run.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5567, in main 2024-09-20T22:49:16.852 INFO:teuthology.orchestra.run.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-20T22:49:16.852 INFO:teuthology.orchestra.run.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-20T22:49:16.852 INFO:teuthology.orchestra.run.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-20T22:49:16.853 INFO:teuthology.orchestra.run.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1088, in deploy_daemon 2024-09-20T22:49:16.853 INFO:teuthology.orchestra.run.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1224, in deploy_daemon_units 2024-09-20T22:49:16.853 INFO:teuthology.orchestra.run.smithi115.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-20T22:49:16.853 INFO:teuthology.orchestra.run.smithi115.stdout:RuntimeError: Failed command: systemctl start ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp: Job for ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service failed because the control process exited with error code. 2024-09-20T22:49:16.853 INFO:teuthology.orchestra.run.smithi115.stdout:See "systemctl status ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" and "journalctl -xeu ceph-9fe0f662-779f-11ef-baf5-efdc52797490@iscsi.foo.smithi115.hwguxp.service" for details. 2024-09-20T22:49:17.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:16 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/3264514666' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:49:17.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:16 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/3264514666' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:49:17.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:16 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/3264514666' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T22:49:17.552 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-09-20T22:49:18.124 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:17 smithi115 ceph-mon[64113]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:18.124 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:17 smithi115 ceph-mon[66621]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:18.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:17 smithi153 ceph-mon[56592]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:19.122 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:18 smithi115 ceph-mon[64113]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:19.122 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:18 smithi115 ceph-mon[66621]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:19.180 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:18 smithi153 ceph-mon[56592]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:19.642 INFO:teuthology.orchestra.run.smithi115.stdout:true 2024-09-20T22:49:20.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:19 smithi153 ceph-mon[56592]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:20.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:19 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/1140055795' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:49:20.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:19 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:20.310 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:19 smithi115 ceph-mon[64113]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:20.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:19 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/1140055795' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:49:20.311 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:19 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:20.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:19 smithi115 ceph-mon[66621]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:20.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:19 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/1140055795' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:49:20.311 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:19 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:20.332 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-09-20T22:49:22.018 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:21 smithi115 ceph-mon[66621]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:22.019 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:21 smithi115 ceph-mon[64113]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:22.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:21 smithi153 ceph-mon[56592]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:22.554 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)" 2024-09-20T22:49:23.164 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:22 smithi115 ceph-mon[64113]: from='client.? 172.21.15.115:0/836996936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:49:23.165 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:22 smithi115 ceph-mon[66621]: from='client.? 172.21.15.115:0/836996936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:49:23.166 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-09-20T22:49:23.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:22 smithi153 ceph-mon[56592]: from='client.? 172.21.15.115:0/836996936' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T22:49:24.133 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:23 smithi115 ceph-mon[64113]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:24.133 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:23 smithi115 ceph-mon[66621]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:24.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:23 smithi153 ceph-mon[56592]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:25.229 INFO:teuthology.orchestra.run.smithi115.stdout:osd 8 115s ago - 2024-09-20T22:49:25.573 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:25 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:49:25] "GET /metrics HTTP/1.1" 200 38804 "" "Prometheus/2.43.0" 2024-09-20T22:49:25.901 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T22:49:25.911 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi115.front.sepia.ceph.com 2024-09-20T22:49:25.911 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- bash -c 'ceph orch upgrade ls' 2024-09-20T22:49:26.154 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:25 smithi115 ceph-mon[64113]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:26.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:25 smithi115 ceph-mon[66621]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:26.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:25 smithi153 ceph-mon[56592]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:27.252 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:26 smithi115 ceph-mon[64113]: from='client.54793 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:27.253 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:26 smithi115 ceph-mon[66621]: from='client.54793 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:27.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:26 smithi153 ceph-mon[56592]: from='client.54793 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:28.218 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-09-20T22:49:28.218 INFO:teuthology.orchestra.run.smithi115.stdout: "image": "quay.io/ceph/ceph", 2024-09-20T22:49:28.219 INFO:teuthology.orchestra.run.smithi115.stdout: "registry": "quay.io", 2024-09-20T22:49:28.219 INFO:teuthology.orchestra.run.smithi115.stdout: "bare_image": "ceph/ceph", 2024-09-20T22:49:28.219 INFO:teuthology.orchestra.run.smithi115.stdout: "versions": [ 2024-09-20T22:49:28.219 INFO:teuthology.orchestra.run.smithi115.stdout: "19.1.1", 2024-09-20T22:49:28.219 INFO:teuthology.orchestra.run.smithi115.stdout: "19.1.0" 2024-09-20T22:49:28.219 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-09-20T22:49:28.219 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-09-20T22:49:28.219 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:27 smithi115 ceph-mon[64113]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:28.219 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:27 smithi115 ceph-mon[66621]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:28.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:27 smithi153 ceph-mon[56592]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:28.842 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-09-20T22:49:29.146 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:28 smithi115 ceph-mon[64113]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:29.146 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:28 smithi115 ceph-mon[64113]: from='client.54799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:29.147 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:28 smithi115 ceph-mon[66621]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:29.147 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:28 smithi115 ceph-mon[66621]: from='client.54799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:29.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:28 smithi153 ceph-mon[56592]: from='client.44748 -' entity='client.iscsi.foo.smithi115.hwguxp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:29.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:28 smithi153 ceph-mon[56592]: from='client.54799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:30.064 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:29 smithi115 ceph-mon[64113]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:30.064 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:29 smithi115 ceph-mon[66621]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:30.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:29 smithi153 ceph-mon[56592]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:31.137 INFO:teuthology.orchestra.run.smithi115.stdout: "16.2.0", 2024-09-20T22:49:31.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:30 smithi153 ceph-mon[56592]: from='client.44868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:31.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:30 smithi153 ceph-mon[56592]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:31.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:30 smithi115 ceph-mon[64113]: from='client.44868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:31.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:30 smithi115 ceph-mon[64113]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:31.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:30 smithi115 ceph-mon[66621]: from='client.44868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:31.404 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:30 smithi115 ceph-mon[66621]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:49:31.785 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-09-20T22:49:32.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:49:32.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:49:32.081 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:31 smithi115 ceph-mon[64113]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:49:32.082 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:31 smithi115 ceph-mon[64113]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:32.082 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:31 smithi115 ceph-mon[64113]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:49:32.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:49:32.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:49:32.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:31 smithi115 ceph-mon[66621]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:49:32.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:31 smithi115 ceph-mon[66621]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:32.082 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:31 smithi115 ceph-mon[66621]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:49:32.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:49:32.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:49:32.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:31 smithi153 ceph-mon[56592]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:49:32.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:31 smithi153 ceph-mon[56592]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:32.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:31 smithi153 ceph-mon[56592]: from='mgr.25345 172.21.15.115:0/1669647981' entity='mgr.y' 2024-09-20T22:49:34.007 INFO:teuthology.orchestra.run.smithi115.stdout: "v16.2.2", 2024-09-20T22:49:34.007 INFO:teuthology.orchestra.run.smithi115.stdout: "v16.2.2-20210505", 2024-09-20T22:49:34.627 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:34 smithi115 ceph-mon[66621]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:34.628 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:34 smithi115 ceph-mon[66621]: from='client.44874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:34.628 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:34 smithi115 ceph-mon[64113]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:34.628 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:34 smithi115 ceph-mon[64113]: from='client.44874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:34.652 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-20T22:49:34.661 INFO:tasks.cephadm:Teardown begin 2024-09-20T22:49:34.661 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T22:49:34.689 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T22:49:34.722 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-20T22:49:34.723 DEBUG:teuthology.orchestra.run.smithi115:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-20T22:49:34.745 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-20T22:49:34.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:34 smithi153 ceph-mon[56592]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:49:34.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:34 smithi153 ceph-mon[56592]: from='client.44874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:49:34.781 INFO:tasks.cephadm:Stopping all daemons... 2024-09-20T22:49:34.781 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-20T22:49:34.781 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.a 2024-09-20T22:49:34.904 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:34 smithi115 systemd[1]: Stopping Ceph mon.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:49:35.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a[64109]: 2024-09-20T22:49:35.095+0000 7f44bfe57640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:49:35.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a[64109]: 2024-09-20T22:49:35.095+0000 7f44bfe57640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-20T22:49:35.404 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:35 smithi115 podman[90572]: 2024-09-20 22:49:35.229053218 +0000 UTC m=+0.254663836 container died 29137eed6a0a64cd5d1ea780f5d3ca5c47919edf6c4795234f98ce6cec1e73ad (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-9e2ac70) 2024-09-20T22:49:35.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:49:35] ENGINE Bus STOPPING 2024-09-20T22:49:35.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: ::ffff:172.21.15.153 - - [20/Sep/2024:22:49:35] "GET /metrics HTTP/1.1" 200 38804 "" "Prometheus/2.43.0" 2024-09-20T22:49:35.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:49:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:49:35.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:49:35] ENGINE Bus STOPPED 2024-09-20T22:49:35.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:49:35] ENGINE Bus STARTING 2024-09-20T22:49:35.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:49:35] ENGINE Serving on http://:::9283 2024-09-20T22:49:35.405 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:35 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: [20/Sep/2024:22:49:35] ENGINE Bus STARTED 2024-09-20T22:49:35.873 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:35 smithi115 podman[90572]: 2024-09-20 22:49:35.579447021 +0000 UTC m=+0.605057645 container cleanup 29137eed6a0a64cd5d1ea780f5d3ca5c47919edf6c4795234f98ce6cec1e73ad (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-20T22:49:35.874 INFO:journalctl@ceph.mon.a.smithi115.stdout:Sep 20 22:49:35 smithi115 bash[90572]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-a 2024-09-20T22:49:36.001 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.a.service' 2024-09-20T22:49:36.052 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:49:36.052 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-20T22:49:36.052 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-20T22:49:36.052 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.c 2024-09-20T22:49:36.154 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:36 smithi115 systemd[1]: Stopping Ceph mon.c for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:49:36.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:36 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c[66617]: 2024-09-20T22:49:36.341+0000 7f61dd2e4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:49:36.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:36 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c[66617]: 2024-09-20T22:49:36.341+0000 7f61dd2e4640 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-09-20T22:49:36.619 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:36 smithi115 podman[90705]: 2024-09-20 22:49:36.413829681 +0000 UTC m=+0.178556977 container died 887b72a1971ef9bd73d6f669768a983207ba2f23841285044bc882cbdec3cad8 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:49:36.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:36 smithi115 podman[90705]: 2024-09-20 22:49:36.717095094 +0000 UTC m=+0.481822402 container cleanup 887b72a1971ef9bd73d6f669768a983207ba2f23841285044bc882cbdec3cad8 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-20T22:49:36.904 INFO:journalctl@ceph.mon.c.smithi115.stdout:Sep 20 22:49:36 smithi115 bash[90705]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-c 2024-09-20T22:49:37.097 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.c.service' 2024-09-20T22:49:37.132 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:49:37.133 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-20T22:49:37.133 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-20T22:49:37.133 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.b 2024-09-20T22:49:37.448 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:37 smithi153 systemd[1]: Stopping Ceph mon.b for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:49:37.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b[56581]: 2024-09-20T22:49:37.444+0000 7f4a47eae640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:49:37.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b[56581]: 2024-09-20T22:49:37.444+0000 7f4a47eae640 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-09-20T22:49:37.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:37 smithi153 podman[74677]: 2024-09-20 22:49:37.538388397 +0000 UTC m=+0.203361140 container died f4dde7cfd242a82e7d2a316f40e280c08448a302b28725c952f838c583f5d490 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2) 2024-09-20T22:49:38.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:37 smithi153 podman[74677]: 2024-09-20 22:49:37.996587033 +0000 UTC m=+0.661559798 container cleanup f4dde7cfd242a82e7d2a316f40e280c08448a302b28725c952f838c583f5d490 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-20T22:49:38.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Sep 20 22:49:37 smithi153 bash[74677]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mon-b 2024-09-20T22:49:38.429 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mon.b.service' 2024-09-20T22:49:38.474 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:49:38.474 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-20T22:49:38.475 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-20T22:49:38.475 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.y 2024-09-20T22:49:38.764 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:38 smithi115 systemd[1]: Stopping Ceph mgr.y for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:49:39.073 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:38 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:49:38.761+0000 7f25e6112640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:49:39.073 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:38 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y[58982]: 2024-09-20T22:49:38.761+0000 7f25e6112640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-20T22:49:39.073 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:38 smithi115 podman[90823]: 2024-09-20 22:49:38.803738484 +0000 UTC m=+0.144669428 container died 07b943d2934a7a3c2abdfc8ac3bc9e2469d678ab0c32fc25a1c4ddd7fe164db8 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-20T22:49:39.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:39 smithi115 podman[90823]: 2024-09-20 22:49:39.163076524 +0000 UTC m=+0.504007467 container cleanup 07b943d2934a7a3c2abdfc8ac3bc9e2469d678ab0c32fc25a1c4ddd7fe164db8 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-20T22:49:39.404 INFO:journalctl@ceph.mgr.y.smithi115.stdout:Sep 20 22:49:39 smithi115 bash[90823]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-y 2024-09-20T22:49:39.645 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.y.service' 2024-09-20T22:49:39.680 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:49:39.680 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-20T22:49:39.681 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-20T22:49:39.681 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.x 2024-09-20T22:49:40.024 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:49:39 smithi153 systemd[1]: Stopping Ceph mgr.x for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:49:40.278 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:49:40 smithi153 podman[74797]: 2024-09-20 22:49:40.023090773 +0000 UTC m=+0.148295171 container died 5de21dfaf787663865b0d98a8a5e3109583ec16e537441aa5df59dcd6ba933bc (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, ceph=True, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:49:40.761 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:49:40 smithi153 podman[74797]: 2024-09-20 22:49:40.46725828 +0000 UTC m=+0.592462678 container cleanup 5de21dfaf787663865b0d98a8a5e3109583ec16e537441aa5df59dcd6ba933bc (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70) 2024-09-20T22:49:40.761 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Sep 20 22:49:40 smithi153 bash[74797]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-mgr-x 2024-09-20T22:49:40.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@mgr.x.service' 2024-09-20T22:49:40.937 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:49:40.938 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-20T22:49:40.938 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-20T22:49:40.938 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.0 2024-09-20T22:49:41.227 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:40 smithi115 systemd[1]: Stopping Ceph osd.0 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:49:41.654 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:41 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0[69857]: 2024-09-20T22:49:41.225+0000 7f8e8d19e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:49:41.654 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:41 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0[69857]: 2024-09-20T22:49:41.225+0000 7f8e8d19e640 -1 osd.0 155 *** Got signal Terminated *** 2024-09-20T22:49:41.654 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:41 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0[69857]: 2024-09-20T22:49:41.225+0000 7f8e8d19e640 -1 osd.0 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:49:46.870 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:46 smithi115 podman[90943]: 2024-09-20 22:49:46.584094506 +0000 UTC m=+5.460821236 container died 98478f94c84ea77bb9dcf7f430a1c28097a4d1e60aa3ab7a7aa6d7f05af06279 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731) 2024-09-20T22:49:47.154 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:46 smithi115 podman[90943]: 2024-09-20 22:49:46.958977677 +0000 UTC m=+5.835704404 container cleanup 98478f94c84ea77bb9dcf7f430a1c28097a4d1e60aa3ab7a7aa6d7f05af06279 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70) 2024-09-20T22:49:47.154 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:46 smithi115 bash[90943]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0 2024-09-20T22:49:47.644 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:47 smithi115 podman[90972]: 2024-09-20 22:49:47.310297084 +0000 UTC m=+0.715624330 container remove 98478f94c84ea77bb9dcf7f430a1c28097a4d1e60aa3ab7a7aa6d7f05af06279 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:49:47.904 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:47 smithi115 podman[91059]: 2024-09-20 22:49:47.545307617 +0000 UTC m=+0.016234878 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:49:47.904 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:47 smithi115 podman[91059]: 2024-09-20 22:49:47.652739286 +0000 UTC m=+0.123666540 container create aec0e4992cd83a1ea586c2f7dd0ab54073625a74b2b6f013e52674ab8e52a0d1 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, ceph=True, GIT_CLEAN=True) 2024-09-20T22:49:47.905 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:47 smithi115 podman[91059]: 2024-09-20 22:49:47.863780241 +0000 UTC m=+0.334707500 container init aec0e4992cd83a1ea586c2f7dd0ab54073625a74b2b6f013e52674ab8e52a0d1 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-20T22:49:47.905 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:47 smithi115 podman[91059]: 2024-09-20 22:49:47.867128918 +0000 UTC m=+0.338056172 container start aec0e4992cd83a1ea586c2f7dd0ab54073625a74b2b6f013e52674ab8e52a0d1 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=squid-9e2ac70) 2024-09-20T22:49:48.360 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:47 smithi115 podman[91059]: 2024-09-20 22:49:47.938782186 +0000 UTC m=+0.409709448 container attach aec0e4992cd83a1ea586c2f7dd0ab54073625a74b2b6f013e52674ab8e52a0d1 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-20T22:49:48.360 INFO:journalctl@ceph.osd.0.smithi115.stdout:Sep 20 22:49:48 smithi115 podman[91059]: 2024-09-20 22:49:48.057699301 +0000 UTC m=+0.528626562 container died aec0e4992cd83a1ea586c2f7dd0ab54073625a74b2b6f013e52674ab8e52a0d1 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-20T22:49:48.696 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.0.service' 2024-09-20T22:49:48.736 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:49:48.736 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-20T22:49:48.736 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-20T22:49:48.736 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.1 2024-09-20T22:49:49.065 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:48 smithi115 systemd[1]: Stopping Ceph osd.1 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:49:49.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:49 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1[72420]: 2024-09-20T22:49:49.063+0000 7f052d720640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:49:49.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:49 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1[72420]: 2024-09-20T22:49:49.063+0000 7f052d720640 -1 osd.1 155 *** Got signal Terminated *** 2024-09-20T22:49:49.404 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:49 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1[72420]: 2024-09-20T22:49:49.063+0000 7f052d720640 -1 osd.1 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:49:54.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:54 smithi115 podman[91162]: 2024-09-20 22:49:54.439365031 +0000 UTC m=+5.474095843 container died 4c2f3b5efcd070a74e35cc2720476e854e4c07d8c1ea368aa5858825d745a456 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-20T22:49:54.655 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:49:54 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:49:54.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi115.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused" 2024-09-20T22:49:54.655 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:49:54 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:49:54.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-09-20T22:49:55.064 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:54 smithi115 podman[91162]: 2024-09-20 22:49:54.786885026 +0000 UTC m=+5.821615827 container cleanup 4c2f3b5efcd070a74e35cc2720476e854e4c07d8c1ea368aa5858825d745a456 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:49:55.064 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:54 smithi115 bash[91162]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1 2024-09-20T22:49:55.279 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:49:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:49:54.807Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.115:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.115:8765: connect: connection refused" 2024-09-20T22:49:55.279 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:49:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:49:54.809Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.115:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.115:8765: connect: connection refused" 2024-09-20T22:49:55.279 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:49:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:49:54.809Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.115:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.115:8765: connect: connection refused" 2024-09-20T22:49:55.279 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:49:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:49:54.810Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.115:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.115:8765: connect: connection refused" 2024-09-20T22:49:55.279 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:49:54 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:49:54.811Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.115:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.115:8765: connect: connection refused" 2024-09-20T22:49:55.401 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:55 smithi115 podman[91178]: 2024-09-20 22:49:55.063087563 +0000 UTC m=+0.621067976 container remove 4c2f3b5efcd070a74e35cc2720476e854e4c07d8c1ea368aa5858825d745a456 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True) 2024-09-20T22:49:55.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:55 smithi115 podman[91264]: 2024-09-20 22:49:55.303191597 +0000 UTC m=+0.015814354 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:49:55.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:55 smithi115 podman[91264]: 2024-09-20 22:49:55.40574823 +0000 UTC m=+0.118371021 container create 702738c170d0e449c87607c61cd957ecebb03dc1a14e22e49bd61c8288d5fa61 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2) 2024-09-20T22:49:55.654 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:55 smithi115 podman[91264]: 2024-09-20 22:49:55.649913801 +0000 UTC m=+0.362536556 container init 702738c170d0e449c87607c61cd957ecebb03dc1a14e22e49bd61c8288d5fa61 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-20T22:49:56.129 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:55 smithi115 podman[91264]: 2024-09-20 22:49:55.653270377 +0000 UTC m=+0.365893138 container start 702738c170d0e449c87607c61cd957ecebb03dc1a14e22e49bd61c8288d5fa61 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:49:56.130 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:55 smithi115 podman[91264]: 2024-09-20 22:49:55.724928701 +0000 UTC m=+0.437551451 container attach 702738c170d0e449c87607c61cd957ecebb03dc1a14e22e49bd61c8288d5fa61 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-20T22:49:56.130 INFO:journalctl@ceph.osd.1.smithi115.stdout:Sep 20 22:49:55 smithi115 podman[91264]: 2024-09-20 22:49:55.842158887 +0000 UTC m=+0.554781634 container died 702738c170d0e449c87607c61cd957ecebb03dc1a14e22e49bd61c8288d5fa61 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:49:56.414 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.1.service' 2024-09-20T22:49:56.455 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:49:56.455 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-20T22:49:56.455 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-20T22:49:56.455 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.2 2024-09-20T22:49:56.654 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:49:56 smithi115 systemd[1]: Stopping Ceph osd.2 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:49:57.154 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:49:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2[74986]: 2024-09-20T22:49:56.772+0000 7f7761b60640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:49:57.154 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:49:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2[74986]: 2024-09-20T22:49:56.772+0000 7f7761b60640 -1 osd.2 155 *** Got signal Terminated *** 2024-09-20T22:49:57.155 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:49:56 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2[74986]: 2024-09-20T22:49:56.772+0000 7f7761b60640 -1 osd.2 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:50:02.558 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:50:02 smithi115 podman[91367]: 2024-09-20 22:50:02.254079583 +0000 UTC m=+5.582587218 container died b90ea318f71233d5c74364f4f4ed199701d50c7250dae89008c4eaeed5f55b64 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:50:02.875 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:50:02 smithi115 podman[91367]: 2024-09-20 22:50:02.631315946 +0000 UTC m=+5.959823573 container cleanup b90ea318f71233d5c74364f4f4ed199701d50c7250dae89008c4eaeed5f55b64 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-20T22:50:02.875 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:50:02 smithi115 bash[91367]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2 2024-09-20T22:50:03.154 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:50:02 smithi115 podman[91386]: 2024-09-20 22:50:02.874021603 +0000 UTC m=+0.618157236 container remove b90ea318f71233d5c74364f4f4ed199701d50c7250dae89008c4eaeed5f55b64 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-20T22:50:03.486 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:50:03 smithi115 podman[91473]: 2024-09-20 22:50:03.183207771 +0000 UTC m=+0.105700330 container create 81cb2420b06d8869608cb93210c37119b7dc23a38517efb934153e4ad13d6c8e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-20T22:50:03.486 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:50:03 smithi115 podman[91473]: 2024-09-20 22:50:03.093398177 +0000 UTC m=+0.015890668 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:50:03.487 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:50:03 smithi115 podman[91473]: 2024-09-20 22:50:03.402143701 +0000 UTC m=+0.324636196 container init 81cb2420b06d8869608cb93210c37119b7dc23a38517efb934153e4ad13d6c8e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-09-20T22:50:03.487 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:50:03 smithi115 podman[91473]: 2024-09-20 22:50:03.405335924 +0000 UTC m=+0.327828414 container start 81cb2420b06d8869608cb93210c37119b7dc23a38517efb934153e4ad13d6c8e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-20T22:50:03.487 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:50:03 smithi115 podman[91473]: 2024-09-20 22:50:03.485453289 +0000 UTC m=+0.407945792 container attach 81cb2420b06d8869608cb93210c37119b7dc23a38517efb934153e4ad13d6c8e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-20T22:50:03.890 INFO:journalctl@ceph.osd.2.smithi115.stdout:Sep 20 22:50:03 smithi115 podman[91473]: 2024-09-20 22:50:03.596059316 +0000 UTC m=+0.518551810 container died 81cb2420b06d8869608cb93210c37119b7dc23a38517efb934153e4ad13d6c8e (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-20T22:50:04.184 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.2.service' 2024-09-20T22:50:04.222 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:50:04.223 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-20T22:50:04.223 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-20T22:50:04.223 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.3 2024-09-20T22:50:04.544 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:04 smithi115 systemd[1]: Stopping Ceph osd.3 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:50:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:04.242Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi115.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-09-20T22:50:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:04.243Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi115.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused" 2024-09-20T22:50:04.544 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:04.244Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-09-20T22:50:04.904 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3[77460]: 2024-09-20T22:50:04.541+0000 7f65a70e5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:50:04.904 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3[77460]: 2024-09-20T22:50:04.541+0000 7f65a70e5640 -1 osd.3 155 *** Got signal Terminated *** 2024-09-20T22:50:04.904 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:04 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3[77460]: 2024-09-20T22:50:04.541+0000 7f65a70e5640 -1 osd.3 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:50:07.528 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:07 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:07.204+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:08.278 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:08 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:08.012+0000 7fbe3e019640 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.477107+0000 front 2024-09-20T22:49:43.477061+0000 (oldest deadline 2024-09-20T22:50:07.576815+0000) 2024-09-20T22:50:08.279 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:08 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:08.221+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:08.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:08 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:08.198+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:08.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:08 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:08.444+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:09.154 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:08 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3[77460]: 2024-09-20T22:50:08.829+0000 7f65a36fb640 -1 osd.3 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:44.853831+0000 front 2024-09-20T22:49:44.853902+0000 (oldest deadline 2024-09-20T22:50:08.254431+0000) 2024-09-20T22:50:09.279 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:09.247+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:09.279 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:09.016+0000 7fbe3e019640 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.477107+0000 front 2024-09-20T22:49:43.477061+0000 (oldest deadline 2024-09-20T22:50:07.576815+0000) 2024-09-20T22:50:09.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:09.164+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:09.778 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:09 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:09.452+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:10.227 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:09 smithi115 podman[91575]: 2024-09-20 22:50:09.954687074 +0000 UTC m=+5.507766205 container died 9eec1f15261b4c6f1e11e67e555e630d49f083f7502577fe21a3ca2efcc9a888 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:50:10.435 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:10 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:10.187+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:10.435 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:10 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:10.037+0000 7fbe3e019640 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.477107+0000 front 2024-09-20T22:49:43.477061+0000 (oldest deadline 2024-09-20T22:50:07.576815+0000) 2024-09-20T22:50:10.435 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:10 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:10.251+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:10.477 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:10 smithi115 podman[91575]: 2024-09-20 22:50:10.299657436 +0000 UTC m=+5.852736560 container cleanup 9eec1f15261b4c6f1e11e67e555e630d49f083f7502577fe21a3ca2efcc9a888 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-20T22:50:10.477 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:10 smithi115 bash[91575]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3 2024-09-20T22:50:10.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:10 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:10.433+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:10.904 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:10 smithi115 podman[91590]: 2024-09-20 22:50:10.475712782 +0000 UTC m=+0.516032217 container remove 9eec1f15261b4c6f1e11e67e555e630d49f083f7502577fe21a3ca2efcc9a888 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70) 2024-09-20T22:50:11.213 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:10 smithi115 podman[91679]: 2024-09-20 22:50:10.823839703 +0000 UTC m=+0.015738985 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:50:11.213 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:10 smithi115 podman[91679]: 2024-09-20 22:50:10.951369365 +0000 UTC m=+0.143268635 container create 117e98ced0242236b073806ac837d28a46090b3d67eb64a36e3a3d5188f01ce3 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70) 2024-09-20T22:50:11.213 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:11 smithi115 podman[91679]: 2024-09-20 22:50:11.212159379 +0000 UTC m=+0.404058653 container init 117e98ced0242236b073806ac837d28a46090b3d67eb64a36e3a3d5188f01ce3 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-deactivate, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True) 2024-09-20T22:50:11.449 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:11 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:11.043+0000 7fbe3e019640 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.477107+0000 front 2024-09-20T22:49:43.477061+0000 (oldest deadline 2024-09-20T22:50:07.576815+0000) 2024-09-20T22:50:11.450 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:11 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:11.238+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:11.450 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:11 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:11.235+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:11.654 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:11 smithi115 podman[91679]: 2024-09-20 22:50:11.215508577 +0000 UTC m=+0.407407849 container start 117e98ced0242236b073806ac837d28a46090b3d67eb64a36e3a3d5188f01ce3 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-deactivate, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-20T22:50:11.654 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:11 smithi115 podman[91679]: 2024-09-20 22:50:11.287425399 +0000 UTC m=+0.479324685 container attach 117e98ced0242236b073806ac837d28a46090b3d67eb64a36e3a3d5188f01ce3 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-20T22:50:11.654 INFO:journalctl@ceph.osd.3.smithi115.stdout:Sep 20 22:50:11 smithi115 podman[91679]: 2024-09-20 22:50:11.40232878 +0000 UTC m=+0.594228064 container died 117e98ced0242236b073806ac837d28a46090b3d67eb64a36e3a3d5188f01ce3 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-3-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-20T22:50:11.778 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:11 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:11.448+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:12.025 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.3.service' 2024-09-20T22:50:12.065 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:50:12.065 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-20T22:50:12.065 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-20T22:50:12.065 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.4 2024-09-20T22:50:12.486 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:12 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:12.225+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:12.487 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:12 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:12.076+0000 7fbe3e019640 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.477107+0000 front 2024-09-20T22:49:43.477061+0000 (oldest deadline 2024-09-20T22:50:07.576815+0000) 2024-09-20T22:50:12.487 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:12 smithi153 systemd[1]: Stopping Ceph osd.4 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:50:12.487 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:12 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:12.191+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:12.778 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:12 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:12.485+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:12.779 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:12 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:12.527+0000 7fbe45a0b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:50:12.779 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:12 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:12.527+0000 7fbe45a0b640 -1 osd.4 155 *** Got signal Terminated *** 2024-09-20T22:50:12.779 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:12 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:12.527+0000 7fbe45a0b640 -1 osd.4 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:50:13.484 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:13.029+0000 7fbe3e019640 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.477107+0000 front 2024-09-20T22:49:43.477061+0000 (oldest deadline 2024-09-20T22:50:07.576815+0000) 2024-09-20T22:50:13.484 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:13.263+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:13.485 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:13.211+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:13.778 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:13 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:13.482+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:14.528 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:14.033+0000 7fbe3e019640 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.477107+0000 front 2024-09-20T22:49:43.477061+0000 (oldest deadline 2024-09-20T22:50:07.576815+0000) 2024-09-20T22:50:14.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:14.266+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:14.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:14.169+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:14.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:14 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:14.243Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi115.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-09-20T22:50:14.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:14 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:14.244Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi115.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused" 2024-09-20T22:50:14.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:14 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:14.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-09-20T22:50:15.028 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:14 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:14.531+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:15.307 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:15 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:15.192+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:15.307 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:15 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:15.051+0000 7fbe3e019640 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.477107+0000 front 2024-09-20T22:49:43.477061+0000 (oldest deadline 2024-09-20T22:50:07.576815+0000) 2024-09-20T22:50:15.578 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:15 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:15.305+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:16.028 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:15 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:15.576+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:16.356 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:16 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:16.206+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:16.356 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:16 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:16.055+0000 7fbe3e019640 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.477107+0000 front 2024-09-20T22:49:43.477061+0000 (oldest deadline 2024-09-20T22:50:07.576815+0000) 2024-09-20T22:50:16.356 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:16 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:16.055+0000 7fbe3e019640 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.877817+0000 front 2024-09-20T22:49:52.877663+0000 (oldest deadline 2024-09-20T22:50:15.777651+0000) 2024-09-20T22:50:16.778 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:16 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:16.590+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:16.779 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:16 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:16.354+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:16.779 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:16 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:16.354+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.028669+0000 front 2024-09-20T22:49:53.028625+0000 (oldest deadline 2024-09-20T22:50:15.928481+0000) 2024-09-20T22:50:17.360 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:17.158+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:17.360 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:17.158+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:17.360 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:17.060+0000 7fbe3e019640 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.477107+0000 front 2024-09-20T22:49:43.477061+0000 (oldest deadline 2024-09-20T22:50:07.576815+0000) 2024-09-20T22:50:17.361 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4[59830]: 2024-09-20T22:50:17.060+0000 7fbe3e019640 -1 osd.4 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.877817+0000 front 2024-09-20T22:49:52.877663+0000 (oldest deadline 2024-09-20T22:50:15.777651+0000) 2024-09-20T22:50:17.778 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:17.359+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:17.779 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:17.359+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.028669+0000 front 2024-09-20T22:49:53.028625+0000 (oldest deadline 2024-09-20T22:50:15.928481+0000) 2024-09-20T22:50:17.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:17.604+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:17.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:17 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:17.604+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:18.278 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:18.180+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:18.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:18.180+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:18.279 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:17 smithi153 podman[74934]: 2024-09-20 22:50:17.984307122 +0000 UTC m=+5.713907014 container died ba7575091ec90d25a515281807273e794daa549960a010e0ee011633a2e8a31b (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-20T22:50:18.622 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:18 smithi153 podman[74934]: 2024-09-20 22:50:18.461237527 +0000 UTC m=+6.190837426 container cleanup ba7575091ec90d25a515281807273e794daa549960a010e0ee011633a2e8a31b (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:50:18.622 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:18 smithi153 bash[74934]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4 2024-09-20T22:50:18.622 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:18.367+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:18.622 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:18.367+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.028669+0000 front 2024-09-20T22:49:53.028625+0000 (oldest deadline 2024-09-20T22:50:15.928481+0000) 2024-09-20T22:50:19.028 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:18 smithi153 podman[74949]: 2024-09-20 22:50:18.642984175 +0000 UTC m=+0.658067080 container remove ba7575091ec90d25a515281807273e794daa549960a010e0ee011633a2e8a31b (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-20T22:50:19.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:18.619+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:19.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:18 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:18.619+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:19.309 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:19.168+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:19.309 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:19.168+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:19.309 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:19 smithi153 podman[75039]: 2024-09-20 22:50:18.979738717 +0000 UTC m=+0.016173140 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:50:19.309 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:19 smithi153 podman[75039]: 2024-09-20 22:50:19.088474963 +0000 UTC m=+0.124909384 container create 5db335d504d75f5fca7a25ab88cab076e25cd5b1e37828c81c77355e8843f3a9 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:50:19.309 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:19 smithi153 podman[75039]: 2024-09-20 22:50:19.307506711 +0000 UTC m=+0.343941136 container init 5db335d504d75f5fca7a25ab88cab076e25cd5b1e37828c81c77355e8843f3a9 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-deactivate, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True) 2024-09-20T22:50:19.622 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:19.358+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:19.622 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:19.358+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.028669+0000 front 2024-09-20T22:49:53.028625+0000 (oldest deadline 2024-09-20T22:50:15.928481+0000) 2024-09-20T22:50:19.622 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:19 smithi153 podman[75039]: 2024-09-20 22:50:19.311831435 +0000 UTC m=+0.348265855 container start 5db335d504d75f5fca7a25ab88cab076e25cd5b1e37828c81c77355e8843f3a9 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2) 2024-09-20T22:50:19.623 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:19 smithi153 podman[75039]: 2024-09-20 22:50:19.382389516 +0000 UTC m=+0.418823936 container attach 5db335d504d75f5fca7a25ab88cab076e25cd5b1e37828c81c77355e8843f3a9 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:50:19.623 INFO:journalctl@ceph.osd.4.smithi153.stdout:Sep 20 22:50:19 smithi153 podman[75039]: 2024-09-20 22:50:19.501069312 +0000 UTC m=+0.537503733 container died 5db335d504d75f5fca7a25ab88cab076e25cd5b1e37828c81c77355e8843f3a9 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-20T22:50:20.028 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:19.620+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:20.028 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:19 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:19.620+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:20.222 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.4.service' 2024-09-20T22:50:20.263 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:50:20.263 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-20T22:50:20.263 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-20T22:50:20.263 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.5 2024-09-20T22:50:20.467 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:20.140+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:20.468 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:20.140+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:20.468 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:20.337+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:20.468 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:20.337+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.028669+0000 front 2024-09-20T22:49:53.028625+0000 (oldest deadline 2024-09-20T22:50:15.928481+0000) 2024-09-20T22:50:20.468 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:20 smithi153 systemd[1]: Stopping Ceph osd.5 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:50:20.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:20.640+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:20.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:20.640+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:20.779 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:20.631+0000 7f0c29c8b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:50:20.779 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:20.631+0000 7f0c29c8b640 -1 osd.5 155 *** Got signal Terminated *** 2024-09-20T22:50:20.779 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:20 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:20.631+0000 7f0c29c8b640 -1 osd.5 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:50:21.420 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:21.356+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:21.420 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:21.356+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.028669+0000 front 2024-09-20T22:49:53.028625+0000 (oldest deadline 2024-09-20T22:50:15.928481+0000) 2024-09-20T22:50:21.421 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:21.156+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:21.421 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:21.156+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:21.778 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:21.599+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:21.779 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:21 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:21.599+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:22.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:22 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:22.405+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:22.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:22 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:22.405+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.028669+0000 front 2024-09-20T22:49:53.028625+0000 (oldest deadline 2024-09-20T22:50:15.928481+0000) 2024-09-20T22:50:22.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:22 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:22.207+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:22.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:22 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:22.207+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:23.028 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:22 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:22.624+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:23.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:22 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:22.624+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:23.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:23.447+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:23.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:23.447+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.028669+0000 front 2024-09-20T22:49:53.028625+0000 (oldest deadline 2024-09-20T22:50:15.928481+0000) 2024-09-20T22:50:23.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:23.252+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:23.530 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:23.252+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:23.530 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:23.252+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:24.028 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:23.670+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:24.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:23 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:23.670+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:24.530 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:24 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:24.441+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:24.530 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:24 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:24.441+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.028669+0000 front 2024-09-20T22:49:53.028625+0000 (oldest deadline 2024-09-20T22:50:15.928481+0000) 2024-09-20T22:50:24.531 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:24 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:24.257+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:24.531 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:24 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:24.257+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:24.531 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:24 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:24.257+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:24.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:24 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:24.244Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi115.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-09-20T22:50:24.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:24 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:24.244Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi115.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused" 2024-09-20T22:50:24.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:24 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:24.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-09-20T22:50:25.028 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:24 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:24.682+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:25.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:24 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:24.682+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:25.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:24 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:24.682+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:50:00.168393+0000 front 2024-09-20T22:50:00.168373+0000 (oldest deadline 2024-09-20T22:50:24.268379+0000) 2024-09-20T22:50:25.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:25 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:25.481+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:46.127652+0000 front 2024-09-20T22:49:46.127591+0000 (oldest deadline 2024-09-20T22:50:07.827394+0000) 2024-09-20T22:50:25.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:25 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5[62127]: 2024-09-20T22:50:25.481+0000 7f0c262a1640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.028669+0000 front 2024-09-20T22:49:53.028625+0000 (oldest deadline 2024-09-20T22:50:15.928481+0000) 2024-09-20T22:50:25.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:25 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:25.254+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:25.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:25 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:25.254+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:25.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:25 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:25.254+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:25.968 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:25 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:25.636+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:25.968 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:25 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:25.636+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:25.968 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:25 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:25.636+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:50:00.168393+0000 front 2024-09-20T22:50:00.168373+0000 (oldest deadline 2024-09-20T22:50:24.268379+0000) 2024-09-20T22:50:26.242 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:25 smithi153 podman[75141]: 2024-09-20 22:50:25.96674459 +0000 UTC m=+5.469192112 container died b1f12957e36e31c47c4359208b48227e1e387336f17ee860e887461d9bc5074b (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:50:26.528 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:26 smithi153 podman[75141]: 2024-09-20 22:50:26.422951537 +0000 UTC m=+5.925399059 container cleanup b1f12957e36e31c47c4359208b48227e1e387336f17ee860e887461d9bc5074b (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-20T22:50:26.529 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:26 smithi153 bash[75141]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5 2024-09-20T22:50:26.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:26 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:26.276+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:26.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:26 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:26.276+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:26.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:26 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:26.276+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:27.028 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:26 smithi153 podman[75167]: 2024-09-20 22:50:26.733035569 +0000 UTC m=+0.767377172 container remove b1f12957e36e31c47c4359208b48227e1e387336f17ee860e887461d9bc5074b (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-20T22:50:27.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:26 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:26.668+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:27.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:26 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:26.668+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:27.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:26 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:26.668+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:50:00.168393+0000 front 2024-09-20T22:50:00.168373+0000 (oldest deadline 2024-09-20T22:50:24.268379+0000) 2024-09-20T22:50:27.362 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:27 smithi153 podman[75259]: 2024-09-20 22:50:26.99176251 +0000 UTC m=+0.015979854 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:50:27.363 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:27 smithi153 podman[75259]: 2024-09-20 22:50:27.125527459 +0000 UTC m=+0.149744782 container create 3c39963ae010d17e863145391e089bfe7a6b7ab8518009e23c4433cd61260077 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-20T22:50:27.363 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:27.302+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:27.363 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:27.302+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:27.363 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:27.302+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:27.668 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:27 smithi153 podman[75259]: 2024-09-20 22:50:27.361296048 +0000 UTC m=+0.385513383 container init 3c39963ae010d17e863145391e089bfe7a6b7ab8518009e23c4433cd61260077 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-20T22:50:27.668 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:27 smithi153 podman[75259]: 2024-09-20 22:50:27.365459238 +0000 UTC m=+0.389676575 container start 3c39963ae010d17e863145391e089bfe7a6b7ab8518009e23c4433cd61260077 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-20T22:50:27.668 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:27 smithi153 podman[75259]: 2024-09-20 22:50:27.452994799 +0000 UTC m=+0.477212122 container attach 3c39963ae010d17e863145391e089bfe7a6b7ab8518009e23c4433cd61260077 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70) 2024-09-20T22:50:27.668 INFO:journalctl@ceph.osd.5.smithi153.stdout:Sep 20 22:50:27 smithi153 podman[75259]: 2024-09-20 22:50:27.563595696 +0000 UTC m=+0.587813035 container died 3c39963ae010d17e863145391e089bfe7a6b7ab8518009e23c4433cd61260077 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-09-20T22:50:28.028 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:27.666+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:28.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:27.666+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:28.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:27 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:27.666+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:50:00.168393+0000 front 2024-09-20T22:50:00.168373+0000 (oldest deadline 2024-09-20T22:50:24.268379+0000) 2024-09-20T22:50:28.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.5.service' 2024-09-20T22:50:28.271 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:50:28.271 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-20T22:50:28.271 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-20T22:50:28.272 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.6 2024-09-20T22:50:28.529 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:28 smithi153 systemd[1]: Stopping Ceph osd.6 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:50:28.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:28.325+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:28.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:28.325+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:28.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:28.325+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:29.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:28.609+0000 7fc5d5eb9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:50:29.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:28.609+0000 7fc5d5eb9640 -1 osd.6 155 *** Got signal Terminated *** 2024-09-20T22:50:29.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:28.609+0000 7fc5d5eb9640 -1 osd.6 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:50:29.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:28.664+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:29.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:28.664+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:29.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:28 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:28.664+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:50:00.168393+0000 front 2024-09-20T22:50:00.168373+0000 (oldest deadline 2024-09-20T22:50:24.268379+0000) 2024-09-20T22:50:29.633 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:29.327+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:29.634 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:29.327+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:29.634 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:29.327+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:30.028 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:29.631+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:30.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:29.631+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:30.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:29 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:29.631+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:50:00.168393+0000 front 2024-09-20T22:50:00.168373+0000 (oldest deadline 2024-09-20T22:50:24.268379+0000) 2024-09-20T22:50:30.616 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:30 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:30.333+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:30.616 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:30 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:30.333+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:30.616 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:30 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:30.333+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:30.928 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:30 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:30.614+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:30.928 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:30 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:30.614+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:30.928 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:30 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:30.614+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:50:00.168393+0000 front 2024-09-20T22:50:00.168373+0000 (oldest deadline 2024-09-20T22:50:24.268379+0000) 2024-09-20T22:50:31.635 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:31 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:31.283+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:31.635 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:31 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:31.283+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:31.635 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:31 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:31.283+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:32.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:31 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:31.633+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:32.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:31 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:31.633+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:32.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:31 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:31.633+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:50:00.168393+0000 front 2024-09-20T22:50:00.168373+0000 (oldest deadline 2024-09-20T22:50:24.268379+0000) 2024-09-20T22:50:32.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:31 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:31.633+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6830 osd.3 since back 2024-09-20T22:50:08.269943+0000 front 2024-09-20T22:50:08.269918+0000 (oldest deadline 2024-09-20T22:50:31.169687+0000) 2024-09-20T22:50:32.528 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:32.236+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:32.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:32.236+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:32.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:32.236+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:33.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:32.593+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:33.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:32.593+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:33.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:32.593+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:50:00.168393+0000 front 2024-09-20T22:50:00.168373+0000 (oldest deadline 2024-09-20T22:50:24.268379+0000) 2024-09-20T22:50:33.029 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:32 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:32.593+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6830 osd.3 since back 2024-09-20T22:50:08.269943+0000 front 2024-09-20T22:50:08.269918+0000 (oldest deadline 2024-09-20T22:50:31.169687+0000) 2024-09-20T22:50:33.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:33.238+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:33.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:33.238+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:33.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:33.238+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:33.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:33.238+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6830 osd.3 since back 2024-09-20T22:50:07.857727+0000 front 2024-09-20T22:50:07.857760+0000 (oldest deadline 2024-09-20T22:50:32.557572+0000) 2024-09-20T22:50:33.954 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:33.553+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:43.365822+0000 front 2024-09-20T22:49:43.365729+0000 (oldest deadline 2024-09-20T22:50:07.465603+0000) 2024-09-20T22:50:33.955 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:33.553+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:52.666749+0000 front 2024-09-20T22:49:52.666876+0000 (oldest deadline 2024-09-20T22:50:17.366632+0000) 2024-09-20T22:50:33.955 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:33.553+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:50:00.168393+0000 front 2024-09-20T22:50:00.168373+0000 (oldest deadline 2024-09-20T22:50:24.268379+0000) 2024-09-20T22:50:33.955 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:33 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6[64421]: 2024-09-20T22:50:33.553+0000 7fc5ce4c7640 -1 osd.6 155 heartbeat_check: no reply from 172.21.15.115:6830 osd.3 since back 2024-09-20T22:50:08.269943+0000 front 2024-09-20T22:50:08.269918+0000 (oldest deadline 2024-09-20T22:50:31.169687+0000) 2024-09-20T22:50:34.229 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:33 smithi153 podman[75360]: 2024-09-20 22:50:33.953468964 +0000 UTC m=+5.458678598 container died c7f1c8d06b01a3fba233ff66d263925b7af2f6a3445a78cbce59383846e5ad08 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-20T22:50:34.229 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:34 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:34.198+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:34.229 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:34 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:34.198+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:34.229 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:34 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:34.198+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:34.229 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:34 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:34.198+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6830 osd.3 since back 2024-09-20T22:50:07.857727+0000 front 2024-09-20T22:50:07.857760+0000 (oldest deadline 2024-09-20T22:50:32.557572+0000) 2024-09-20T22:50:34.528 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:34 smithi153 podman[75360]: 2024-09-20 22:50:34.417938208 +0000 UTC m=+5.923147835 container cleanup c7f1c8d06b01a3fba233ff66d263925b7af2f6a3445a78cbce59383846e5ad08 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-20T22:50:34.528 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:34 smithi153 bash[75360]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6 2024-09-20T22:50:34.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:34.245Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi115.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-09-20T22:50:34.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:34.245Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi115.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused" 2024-09-20T22:50:34.654 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:34 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:34.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-09-20T22:50:34.982 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:34 smithi153 podman[75375]: 2024-09-20 22:50:34.660898326 +0000 UTC m=+0.705017747 container remove c7f1c8d06b01a3fba233ff66d263925b7af2f6a3445a78cbce59383846e5ad08 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-20T22:50:35.278 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:35 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:35.180+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:35.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:35 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:35.180+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:35.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:35 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:35.180+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:35.279 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:35 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:35.180+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6830 osd.3 since back 2024-09-20T22:50:07.857727+0000 front 2024-09-20T22:50:07.857760+0000 (oldest deadline 2024-09-20T22:50:32.557572+0000) 2024-09-20T22:50:35.279 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:34 smithi153 podman[75465]: 2024-09-20 22:50:34.884262248 +0000 UTC m=+0.016046276 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:50:35.279 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:34 smithi153 podman[75465]: 2024-09-20 22:50:34.995020842 +0000 UTC m=+0.126804906 container create 27888de70eb16dbc74eae9d8c6371a056332a716f1e839c8d50781504bda15a4 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux ) 2024-09-20T22:50:35.279 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:35 smithi153 podman[75465]: 2024-09-20 22:50:35.197212128 +0000 UTC m=+0.328996154 container init 27888de70eb16dbc74eae9d8c6371a056332a716f1e839c8d50781504bda15a4 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-deactivate, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-20T22:50:35.279 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:35 smithi153 podman[75465]: 2024-09-20 22:50:35.200492849 +0000 UTC m=+0.332276868 container start 27888de70eb16dbc74eae9d8c6371a056332a716f1e839c8d50781504bda15a4 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-deactivate, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70) 2024-09-20T22:50:35.685 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:35 smithi153 podman[75465]: 2024-09-20 22:50:35.280423122 +0000 UTC m=+0.412207140 container attach 27888de70eb16dbc74eae9d8c6371a056332a716f1e839c8d50781504bda15a4 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-20T22:50:35.685 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:35 smithi153 podman[75465]: 2024-09-20 22:50:35.396523041 +0000 UTC m=+0.528307067 container died 27888de70eb16dbc74eae9d8c6371a056332a716f1e839c8d50781504bda15a4 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-20T22:50:35.952 INFO:journalctl@ceph.osd.6.smithi153.stdout:Sep 20 22:50:35 smithi153 podman[75489]: 2024-09-20 22:50:35.925314413 +0000 UTC m=+0.519304683 container remove 27888de70eb16dbc74eae9d8c6371a056332a716f1e839c8d50781504bda15a4 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2) 2024-09-20T22:50:35.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.6.service' 2024-09-20T22:50:35.994 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:50:35.994 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-20T22:50:35.994 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-20T22:50:35.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.7 2024-09-20T22:50:36.228 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:36 smithi153 systemd[1]: Stopping Ceph osd.7 for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:50:36.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:36 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:36.225+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:36.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:36 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:36.225+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:36.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:36 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:36.225+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:36.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:36 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:36.225+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6830 osd.3 since back 2024-09-20T22:50:07.857727+0000 front 2024-09-20T22:50:07.857760+0000 (oldest deadline 2024-09-20T22:50:32.557572+0000) 2024-09-20T22:50:36.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:36 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:36.319+0000 7f8599db2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:50:36.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:36 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:36.319+0000 7f8599db2640 -1 osd.7 155 *** Got signal Terminated *** 2024-09-20T22:50:36.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:36 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:36.319+0000 7f8599db2640 -1 osd.7 155 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:50:37.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:37.221+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:37.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:37.221+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:37.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:37.221+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:37.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:37 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:37.221+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6830 osd.3 since back 2024-09-20T22:50:07.857727+0000 front 2024-09-20T22:50:07.857760+0000 (oldest deadline 2024-09-20T22:50:32.557572+0000) 2024-09-20T22:50:38.528 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:38.186+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:38.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:38.186+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:38.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:38.186+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:38.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:38 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:38.186+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6830 osd.3 since back 2024-09-20T22:50:07.857727+0000 front 2024-09-20T22:50:07.857760+0000 (oldest deadline 2024-09-20T22:50:32.557572+0000) 2024-09-20T22:50:39.528 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:39.177+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:39.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:39.177+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:39.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:39.177+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:39.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:39 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:39.177+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6830 osd.3 since back 2024-09-20T22:50:07.857727+0000 front 2024-09-20T22:50:07.857760+0000 (oldest deadline 2024-09-20T22:50:32.557572+0000) 2024-09-20T22:50:40.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:40 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:40.142+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:40.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:40 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:40.142+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:40.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:40 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:40.142+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:40.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:40 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:40.142+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6830 osd.3 since back 2024-09-20T22:50:07.857727+0000 front 2024-09-20T22:50:07.857760+0000 (oldest deadline 2024-09-20T22:50:32.557572+0000) 2024-09-20T22:50:40.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:40 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:40.142+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.153:6804 osd.4 since back 2024-09-20T22:50:14.258593+0000 front 2024-09-20T22:50:14.258479+0000 (oldest deadline 2024-09-20T22:50:39.558542+0000) 2024-09-20T22:50:41.528 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:41 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:41.121+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6806 osd.0 since back 2024-09-20T22:49:45.754873+0000 front 2024-09-20T22:49:45.754912+0000 (oldest deadline 2024-09-20T22:50:06.854788+0000) 2024-09-20T22:50:41.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:41 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:41.121+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6814 osd.1 since back 2024-09-20T22:49:53.256212+0000 front 2024-09-20T22:49:53.256153+0000 (oldest deadline 2024-09-20T22:50:16.656558+0000) 2024-09-20T22:50:41.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:41 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:41.121+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6822 osd.2 since back 2024-09-20T22:49:56.656999+0000 front 2024-09-20T22:49:56.656954+0000 (oldest deadline 2024-09-20T22:50:22.556918+0000) 2024-09-20T22:50:41.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:41 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:41.121+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.115:6830 osd.3 since back 2024-09-20T22:50:07.857727+0000 front 2024-09-20T22:50:07.857760+0000 (oldest deadline 2024-09-20T22:50:32.557572+0000) 2024-09-20T22:50:41.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:41 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7[66715]: 2024-09-20T22:50:41.121+0000 7f85923c0640 -1 osd.7 155 heartbeat_check: no reply from 172.21.15.153:6804 osd.4 since back 2024-09-20T22:50:14.258593+0000 front 2024-09-20T22:50:14.258479+0000 (oldest deadline 2024-09-20T22:50:39.558542+0000) 2024-09-20T22:50:42.122 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:41 smithi153 podman[75569]: 2024-09-20 22:50:41.848211369 +0000 UTC m=+5.632097455 container died ca686c937f4206cbb65def3b2f4acfc04742e0e391f65ae08ab4ffdfd0c21026 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-20T22:50:42.497 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:42 smithi153 podman[75569]: 2024-09-20 22:50:42.219693958 +0000 UTC m=+6.003580043 container cleanup ca686c937f4206cbb65def3b2f4acfc04742e0e391f65ae08ab4ffdfd0c21026 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-09-20T22:50:42.497 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:42 smithi153 bash[75569]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7 2024-09-20T22:50:42.778 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:42 smithi153 podman[75585]: 2024-09-20 22:50:42.495827844 +0000 UTC m=+0.646791052 container remove ca686c937f4206cbb65def3b2f4acfc04742e0e391f65ae08ab4ffdfd0c21026 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-20T22:50:43.158 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:42 smithi153 podman[75679]: 2024-09-20 22:50:42.739427112 +0000 UTC m=+0.015958019 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:50:43.158 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:42 smithi153 podman[75679]: 2024-09-20 22:50:42.863403081 +0000 UTC m=+0.139933987 container create 819d02cd444b0d407a93cb4495fdeff837ca2f94c97437213994c3fabca5fa41 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70) 2024-09-20T22:50:43.159 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:43 smithi153 podman[75679]: 2024-09-20 22:50:43.082117386 +0000 UTC m=+0.358648295 container init 819d02cd444b0d407a93cb4495fdeff837ca2f94c97437213994c3fabca5fa41 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-deactivate, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-20T22:50:43.159 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:43 smithi153 podman[75679]: 2024-09-20 22:50:43.085322363 +0000 UTC m=+0.361853280 container start 819d02cd444b0d407a93cb4495fdeff837ca2f94c97437213994c3fabca5fa41 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-20T22:50:43.159 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:43 smithi153 podman[75679]: 2024-09-20 22:50:43.15725563 +0000 UTC m=+0.433786542 container attach 819d02cd444b0d407a93cb4495fdeff837ca2f94c97437213994c3fabca5fa41 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-deactivate, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-20T22:50:43.529 INFO:journalctl@ceph.osd.7.smithi153.stdout:Sep 20 22:50:43 smithi153 podman[75679]: 2024-09-20 22:50:43.270231515 +0000 UTC m=+0.546762430 container died 819d02cd444b0d407a93cb4495fdeff837ca2f94c97437213994c3fabca5fa41 (image=quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-20T22:50:43.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@osd.7.service' 2024-09-20T22:50:43.913 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:50:43.913 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-20T22:50:43.913 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-20T22:50:43.914 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-9fe0f662-779f-11ef-baf5-efdc52797490@prometheus.a 2024-09-20T22:50:44.240 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:43 smithi153 systemd[1]: Stopping Ceph prometheus.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:50:44.523 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:50:44.238Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T22:50:44.523 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:50:44.238Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-20T22:50:44.524 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:50:44.238Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-20T22:50:44.524 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:50:44.238Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T22:50:44.524 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:50:44.238Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-20T22:50:44.524 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:50:44.238Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-20T22:50:44.524 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:50:44.238Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T22:50:44.524 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:50:44.238Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-20T22:50:44.524 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:50:44.239Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-20T22:50:44.524 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:50:44.239Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T22:50:44.524 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:50:44.239Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-20T22:50:44.524 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a[52672]: ts=2024-09-20T22:50:44.239Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-20T22:50:44.524 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 podman[75781]: 2024-09-20 22:50:44.249111759 +0000 UTC m=+0.116888303 container died cb068a98c110f12c21687974a1203265ec172fb1fd15f6bb6aa78c18b0eec275 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:50:44.565 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:44.246Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi115.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused" 2024-09-20T22:50:44.565 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:44.246Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi115.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.115:8443: connect: connection refused" 2024-09-20T22:50:44.565 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:44 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:44.247Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-09-20T22:50:44.778 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 podman[75781]: 2024-09-20 22:50:44.621389425 +0000 UTC m=+0.489165979 container cleanup cb068a98c110f12c21687974a1203265ec172fb1fd15f6bb6aa78c18b0eec275 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:50:44.778 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Sep 20 22:50:44 smithi153 bash[75781]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T22:50:45.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9fe0f662-779f-11ef-baf5-efdc52797490@prometheus.a.service' 2024-09-20T22:50:45.076 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:50:45.077 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-20T22:50:45.077 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-20T22:50:46.943 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:50:46 smithi115 systemd[1]: Stopping Ceph node-exporter.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:50:47.225 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:50:46 smithi115 podman[92080]: 2024-09-20 22:50:46.94144118 +0000 UTC m=+0.122940930 container died 807c51f95c4d867cc619f29e585a7edd598269dee82fd1aaad24c2f294ec77af (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:50:47.566 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:50:47 smithi115 podman[92080]: 2024-09-20 22:50:47.313832147 +0000 UTC m=+0.495331893 container cleanup 807c51f95c4d867cc619f29e585a7edd598269dee82fd1aaad24c2f294ec77af (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:50:47.566 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:50:47 smithi115 bash[92080]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a 2024-09-20T22:50:47.904 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:50:47 smithi115 podman[92092]: 2024-09-20 22:50:47.565003412 +0000 UTC m=+0.627273098 container remove 807c51f95c4d867cc619f29e585a7edd598269dee82fd1aaad24c2f294ec77af (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:50:47.904 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:50:47 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-20T22:50:47.904 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:50:47 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-20T22:50:47.904 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:50:47 smithi115 systemd[1]: Stopped Ceph node-exporter.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:50:47.904 INFO:journalctl@ceph.node-exporter.a.smithi115.stdout:Sep 20 22:50:47 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.a.service: Consumed 2.755s CPU time. 2024-09-20T22:50:48.193 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:47 smithi115 systemd[1]: Stopping Ceph alertmanager.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:50:48.193 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:48 smithi115 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a[57841]: ts=2024-09-20T22:50:48.151Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-20T22:50:48.462 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:48 smithi115 podman[92199]: 2024-09-20 22:50:48.192486771 +0000 UTC m=+0.132312348 container died 7b7065a4ae73c148820d85894e397aa051855b49f7420b00aa201e4854d9101e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:50:48.837 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:48 smithi115 podman[92199]: 2024-09-20 22:50:48.568461369 +0000 UTC m=+0.508286946 container cleanup 7b7065a4ae73c148820d85894e397aa051855b49f7420b00aa201e4854d9101e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:50:48.837 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:48 smithi115 bash[92199]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a 2024-09-20T22:50:49.154 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:48 smithi115 podman[92211]: 2024-09-20 22:50:48.836525925 +0000 UTC m=+0.638317007 container remove 7b7065a4ae73c148820d85894e397aa051855b49f7420b00aa201e4854d9101e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:50:49.154 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:48 smithi115 podman[92211]: 2024-09-20 22:50:48.912093157 +0000 UTC m=+0.713884242 volume remove 8f9024ca1710b61dcda82645f5141ac30ab2b65da7ccfb99e59da0a7dbd86367 2024-09-20T22:50:49.154 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:49 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-20T22:50:49.154 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:49 smithi115 systemd[1]: Stopped Ceph alertmanager.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:50:49.154 INFO:journalctl@ceph.alertmanager.a.smithi115.stdout:Sep 20 22:50:49 smithi115 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@alertmanager.a.service: Consumed 1.453s CPU time. 2024-09-20T22:51:23.389 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-20T22:51:25.237 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:51:24 smithi153 systemd[1]: Stopping Ceph node-exporter.b for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:51:25.514 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:51:25 smithi153 podman[76187]: 2024-09-20 22:51:25.265172665 +0000 UTC m=+0.118494530 container died 3f057318ca363453e1220e6a9405f0bac57b29de1cf86494d260005de29eedd7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:51:25.778 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:51:25 smithi153 podman[76187]: 2024-09-20 22:51:25.687927891 +0000 UTC m=+0.541249744 container cleanup 3f057318ca363453e1220e6a9405f0bac57b29de1cf86494d260005de29eedd7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:51:25.779 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:51:25 smithi153 bash[76187]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b 2024-09-20T22:51:26.278 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:51:25 smithi153 podman[76200]: 2024-09-20 22:51:25.930722597 +0000 UTC m=+0.669298349 container remove 3f057318ca363453e1220e6a9405f0bac57b29de1cf86494d260005de29eedd7 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:51:26.278 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:51:25 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-20T22:51:26.279 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:51:26 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-20T22:51:26.279 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:51:26 smithi153 systemd[1]: Stopped Ceph node-exporter.b for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:51:26.279 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Sep 20 22:51:26 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@node-exporter.b.service: Consumed 2.774s CPU time. 2024-09-20T22:51:26.728 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:51:26 smithi153 systemd[1]: Stopping Ceph grafana.a for 9fe0f662-779f-11ef-baf5-efdc52797490... 2024-09-20T22:51:26.728 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:51:26 smithi153 ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a[49991]: t=2024-09-20T22:51:26+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-09-20T22:51:26.728 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:51:26 smithi153 podman[76283]: 2024-09-20 22:51:26.461056921 +0000 UTC m=+0.104329767 container died 568386e3ebfd2c688abb1ec4721008a0b561781daf551ae37817f38d2118b530 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, description=Ceph Grafana Container, distribution-scope=public, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, name=ubi8, io.openshift.expose-services=, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5) 2024-09-20T22:51:27.028 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:51:26 smithi153 podman[76283]: 2024-09-20 22:51:26.808595842 +0000 UTC m=+0.451868722 container cleanup 568386e3ebfd2c688abb1ec4721008a0b561781daf551ae37817f38d2118b530 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a, vcs-type=git, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, release=236.1648460182, name=ubi8, description=Ceph Grafana Container, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc.) 2024-09-20T22:51:27.029 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:51:26 smithi153 bash[76283]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a 2024-09-20T22:51:27.029 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:51:26 smithi153 bash[76309]: Error: no container with name or ID "ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana.a" found: no such container 2024-09-20T22:51:27.528 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:51:27 smithi153 podman[76295]: 2024-09-20 22:51:27.102039321 +0000 UTC m=+0.641852611 container remove 568386e3ebfd2c688abb1ec4721008a0b561781daf551ae37817f38d2118b530 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-9fe0f662-779f-11ef-baf5-efdc52797490-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-type=git, architecture=x86_64, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , io.openshift.expose-services=, version=8.5, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc.) 2024-09-20T22:51:27.529 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:51:27 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@grafana.a.service: Deactivated successfully. 2024-09-20T22:51:27.529 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:51:27 smithi153 systemd[1]: Stopped Ceph grafana.a for 9fe0f662-779f-11ef-baf5-efdc52797490. 2024-09-20T22:51:27.529 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Sep 20 22:51:27 smithi153 systemd[1]: ceph-9fe0f662-779f-11ef-baf5-efdc52797490@grafana.a.service: Consumed 3.588s CPU time. 2024-09-20T22:51:50.363 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T22:51:50.398 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T22:51:50.425 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-20T22:51:50.427 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913005/remote/smithi115/crash 2024-09-20T22:51:50.428 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/crash -- . 2024-09-20T22:51:50.504 INFO:teuthology.orchestra.run.smithi115.stderr:tar: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-20T22:51:50.504 INFO:teuthology.orchestra.run.smithi115.stderr:tar: Error is not recoverable: exiting now 2024-09-20T22:51:50.506 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913005/remote/smithi153/crash 2024-09-20T22:51:50.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/crash -- . 2024-09-20T22:51:50.549 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-20T22:51:50.549 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-09-20T22:51:50.550 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-20T22:51:50.550 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v CEPHADM_AGENT_DOWN | head -n 1 2024-09-20T22:51:50.581 INFO:tasks.cephadm:Compressing logs... 2024-09-20T22:51:50.581 DEBUG:teuthology.orchestra.run.smithi115:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-20T22:51:50.623 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-20T22:51:50.647 INFO:teuthology.orchestra.run.smithi115.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.0.log 2024-09-20T22:51:50.647 INFO:teuthology.orchestra.run.smithi115.stderr:: No such file or directory 2024-09-20T22:51:50.647 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/tcmu-runner.log 2024-09-20T22:51:50.654 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.1.log 2024-09-20T22:51:50.654 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.0.log: /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-client.rgw.smpl.smithi115.pccgis.log 2024-09-20T22:51:50.654 INFO:teuthology.orchestra.run.smithi115.stderr: 86.3% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/tcmu-runner.log.gz 2024-09-20T22:51:50.654 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-20T22:51:50.654 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-client.rgw.smpl.smithi115.pccgis.log: gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-20T22:51:50.655 INFO:teuthology.orchestra.run.smithi115.stderr: 76.4% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-client.rgw.smpl.smithi115.pccgis.log.gz 2024-09-20T22:51:50.655 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-client.rgw.foo.smithi115.cotebr.log 2024-09-20T22:51:50.655 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mon.c.log 2024-09-20T22:51:50.655 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.audit.log: /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-client.rgw.foo.smithi115.cotebr.log: gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-20T22:51:50.655 INFO:teuthology.orchestra.run.smithi115.stderr: 76.4% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-client.rgw.foo.smithi115.cotebr.log.gz 2024-09-20T22:51:50.655 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mon.a.log 2024-09-20T22:51:50.655 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mon.c.log: /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.2.log 2024-09-20T22:51:50.655 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mgr.y.log 2024-09-20T22:51:50.655 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.2.log: 90.9% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-20T22:51:50.655 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.log 2024-09-20T22:51:50.657 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.7.log 2024-09-20T22:51:50.657 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-09-20T22:51:50.657 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-20T22:51:50.657 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.4.log 2024-09-20T22:51:50.657 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-20T22:51:50.657 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-client.rgw.foo.smithi153.vetrnf.log 2024-09-20T22:51:50.657 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-20T22:51:50.657 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.audit.log: /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-client.rgw.foo.smithi153.vetrnf.log: gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.6.log 2024-09-20T22:51:50.657 INFO:teuthology.orchestra.run.smithi153.stderr: 76.0% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-client.rgw.foo.smithi153.vetrnf.log.gz 2024-09-20T22:51:50.657 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.5.log 2024-09-20T22:51:50.657 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-client.rgw.smpl.smithi153.oznbtc.log 2024-09-20T22:51:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.5.log: gzip -5 83.8% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-20T22:51:50.658 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.log 2024-09-20T22:51:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mgr.x.log 2024-09-20T22:51:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-client.rgw.smpl.smithi153.oznbtc.log: /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.log: 76.4% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-client.rgw.smpl.smithi153.oznbtc.log.gz 2024-09-20T22:51:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mon.b.log 2024-09-20T22:51:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mgr.x.log: 90.7% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-20T22:51:50.658 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 87.8% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-20T22:51:50.658 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/cephadm.log 2024-09-20T22:51:50.659 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mgr.y.log: 94.3% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-20T22:51:50.659 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.3.log 2024-09-20T22:51:50.661 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.log: 91.0% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-20T22:51:50.661 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-20T22:51:50.662 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mon.b.log: /var/log/ceph/cephadm.log: 91.1% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-20T22:51:50.662 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.3.log: 93.2% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-20T22:51:50.674 INFO:teuthology.orchestra.run.smithi153.stderr: 90.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-20T22:51:50.675 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-20T22:51:50.702 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mgr.x.log.gz 2024-09-20T22:51:51.003 INFO:teuthology.orchestra.run.smithi115.stderr: 89.0% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mgr.y.log.gz 2024-09-20T22:51:51.075 INFO:teuthology.orchestra.run.smithi153.stderr: 92.4% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mon.b.log.gz 2024-09-20T22:51:51.120 INFO:teuthology.orchestra.run.smithi115.stderr: 92.2% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mon.c.log.gz 2024-09-20T22:51:51.560 INFO:teuthology.orchestra.run.smithi115.stderr: 91.0% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-mon.a.log.gz 2024-09-20T22:51:51.637 INFO:teuthology.orchestra.run.smithi153.stderr: 93.5% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.6.log.gz 2024-09-20T22:51:51.720 INFO:teuthology.orchestra.run.smithi115.stderr: 93.6% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.2.log.gz 2024-09-20T22:51:52.002 INFO:teuthology.orchestra.run.smithi153.stderr: 93.6% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.5.log.gz 2024-09-20T22:51:52.084 INFO:teuthology.orchestra.run.smithi115.stderr: 93.6% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.1.log.gz 2024-09-20T22:51:52.107 INFO:teuthology.orchestra.run.smithi153.stderr: 93.8% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.4.log.gz 2024-09-20T22:51:52.123 INFO:teuthology.orchestra.run.smithi153.stderr: 93.8% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.7.log.gz 2024-09-20T22:51:52.125 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-20T22:51:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m1.487s 2024-09-20T22:51:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m5.565s 2024-09-20T22:51:52.125 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.236s 2024-09-20T22:51:52.161 INFO:teuthology.orchestra.run.smithi115.stderr: 93.6% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.3.log.gz 2024-09-20T22:51:52.189 INFO:teuthology.orchestra.run.smithi115.stderr: 93.7% -- replaced with /var/log/ceph/9fe0f662-779f-11ef-baf5-efdc52797490/ceph-osd.0.log.gz 2024-09-20T22:51:52.191 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-09-20T22:51:52.191 INFO:teuthology.orchestra.run.smithi115.stderr:real 0m1.554s 2024-09-20T22:51:52.191 INFO:teuthology.orchestra.run.smithi115.stderr:user 0m7.037s 2024-09-20T22:51:52.191 INFO:teuthology.orchestra.run.smithi115.stderr:sys 0m0.294s 2024-09-20T22:51:52.191 INFO:tasks.cephadm:Archiving logs... 2024-09-20T22:51:52.191 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913005/remote/smithi115/log 2024-09-20T22:51:52.193 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-20T22:51:53.097 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913005/remote/smithi153/log 2024-09-20T22:51:53.098 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-20T22:51:53.856 INFO:tasks.cephadm:Removing cluster... 2024-09-20T22:51:53.857 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 --force 2024-09-20T22:51:54.128 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9fe0f662-779f-11ef-baf5-efdc52797490 --force 2024-09-20T22:51:54.401 INFO:tasks.cephadm:Removing cephadm ... 2024-09-20T22:51:54.401 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-20T22:51:54.418 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-20T22:51:54.433 INFO:tasks.cephadm:Teardown complete 2024-09-20T22:51:54.433 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-20T22:51:54.442 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-20T22:51:54.442 DEBUG:teuthology.orchestra.run.smithi115:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T22:51:54.460 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T22:51:54.473 INFO:teuthology.orchestra.run.smithi115.stderr:bash: line 1: ntpq: command not found 2024-09-20T22:51:54.476 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T22:51:54.476 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-09-20T22:51:54.476 INFO:teuthology.orchestra.run.smithi115.stdout:^* hv01.front.sepia.ceph.com 3 6 377 59 -319us[ -317us] +/- 40ms 2024-09-20T22:51:54.477 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 58 -191us[ -191us] +/- 73ms 2024-09-20T22:51:54.477 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 53 +838us[ +838us] +/- 78ms 2024-09-20T22:51:54.477 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:51:54.487 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-09-20T22:51:54.491 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T22:51:54.491 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-09-20T22:51:54.491 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv01.front.sepia.ceph.com 3 6 377 54 -274us[ -272us] +/- 40ms 2024-09-20T22:51:54.491 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 59 -170us[ -168us] +/- 73ms 2024-09-20T22:51:54.491 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +833us[ +836us] +/- 78ms 2024-09-20T22:51:54.491 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:51:54.492 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-20T22:51:54.501 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-20T22:51:54.501 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-20T22:51:54.523 DEBUG:teuthology.orchestra.run.smithi115:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-20T22:51:54.601 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-20T22:51:54.674 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-20T22:51:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:54.702 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-20T22:51:54.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:54.730 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-20T22:51:54.745 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-20T22:51:54.764 INFO:teuthology.task.internal:Duration was 1647.946383 seconds 2024-09-20T22:51:54.765 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-20T22:51:54.799 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-20T22:51:54.799 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-20T22:51:54.801 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-20T22:51:54.838 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T22:51:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T22:51:55.191 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-20T22:51:55.191 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi115.front.sepia.ceph.com 2024-09-20T22:51:55.192 DEBUG:teuthology.orchestra.run.smithi115:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-20T22:51:55.252 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-09-20T22:51:55.252 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-20T22:51:55.287 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-20T22:51:55.287 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-20T22:51:55.294 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-20T22:51:55.355 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-20T22:51:55.356 DEBUG:teuthology.orchestra.run.smithi115:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-20T22:51:55.360 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-20T22:51:55.792 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-20T22:51:55.804 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-20T22:51:55.804 DEBUG:teuthology.orchestra.run.smithi115:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-20T22:51:55.828 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-20T22:51:55.853 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-20T22:51:55.863 DEBUG:teuthology.orchestra.run.smithi115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-20T22:51:55.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-20T22:51:55.892 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = core 2024-09-20T22:51:55.917 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-09-20T22:51:55.957 DEBUG:teuthology.orchestra.run.smithi115:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-20T22:51:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:55.971 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-20T22:51:56.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:56.012 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-20T22:51:56.022 INFO:teuthology.task.internal:Transferring archived files... 2024-09-20T22:51:56.023 DEBUG:teuthology.misc:Transferring archived files from smithi115:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913005/remote/smithi115 2024-09-20T22:51:56.023 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-20T22:51:56.067 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913005/remote/smithi153 2024-09-20T22:51:56.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-20T22:51:56.104 INFO:teuthology.task.internal:Removing archive directory... 2024-09-20T22:51:56.105 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-20T22:51:56.107 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-20T22:51:56.154 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-20T22:51:56.164 INFO:teuthology.task.internal:Not uploading archives. 2024-09-20T22:51:56.164 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-20T22:51:56.174 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-20T22:51:56.174 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-20T22:51:56.176 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-20T22:51:56.189 INFO:teuthology.orchestra.run.smithi115.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 20 22:51 /home/ubuntu/cephtest 2024-09-20T22:51:56.209 INFO:teuthology.orchestra.run.smithi153.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 20 22:51 /home/ubuntu/cephtest 2024-09-20T22:51:56.210 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-20T22:51:56.220 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-20T22:51:56.241 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1647.9463834762573 owner: scheduled_teuthology@teuthology success: true 2024-09-20T22:51:56.241 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-20T22:51:56.312 INFO:teuthology.run:pass